I would like to devote an article to an overview of the APIs provided by different operating systems for tracking changes in a directory. The article appeared as a result of my work on change tracking daemons for the dklab_realsync utility (, github repository) and my own, which I do not want to announce yet.
Windows, ReadDirectoryChangesW
For operating system Windows has a wonderful ReadDirectoryChangesW function that returns a set of changes for a directory, including a flag to work recursively (bWatchSubtree). Thus, the implementation of tracking changes in a directory is not difficult, and in the same dklab_realsync implementation takes 80 lines of code or 3.5 KB. Interestingly, Windows even supports these events via SMB!However, there are certain pitfalls:
- the final size of the change buffer, after which the event queue will overflow and these events will be lost
- according to the watchdog package documentation, the move event is sent before the changes are visible in the FS
- the buffer size is limited to 64 KB for the network FS
Conclusion: The ReadDirectoryChangesW function makes it easy to learn about all the events in the files, but the event queue may overflow and then you will need to perform a full file system scan. It is also possible to deliver events before they become relevant.
Mac OS X, FSEvents
Mac OS X also has a handy and simple API for keeping track of changes in the file system called FSEvents. Using this API, the simplest daemon implementation is 50 lines of code, or 1.8 kb. The queue cannot overflow (!), But a full scan may still be required if the fseventsd daemon crashes. It is worth noting that this API does not provide file-by-file changes prior to version 10.7, it only reports directories where something has changed. Since events don't go anywhere and are logged (FSEvents service stores events in a persistent, per-volume database), directory-specific granularity saves disk space.Conclusion: The FSEvents API for Mac OS X is the most unusual of all such APIs. The queue does not overflow and it is even possible to receive events from the past. However, event granularity is given to directory precision (up to 10.7), which means the daemon is less efficient for file synchronization.
Linux, inotify
In linux vanilla kernel, there is one way to keep track of changes in a directory - inotify. There is good and detailed documentation for this API, but there is no support for recursive change tracking! Also, inotify has a limit on maximum amount objects to follow. The simplest implementation of the daemon already takes 250 lines of code, or 8 kb. A static build using dietlibc is approximately 14kb. Another unpleasant point is that the application itself must maintain the correspondence between the watch descriptor (in our case, it is always a directory) and the name. There is a function inotify_add_watch, which is passed the path to the monitored directory, but there is no reverse - inotify_get_path, which would return this very path according to the passed descriptor. Events contain only watch descriptor and the relative path to the changed file inside the directory.The pitfalls of recursive directory tracking via inotify:
- Queue overflow (queue length is set in / proc / sys / fs / inotify / max_queued_events)
- Limit on the maximum number of tracking objects (set in / proc / sys / fs / inotify / max_user_watches)
- Lack of recursive directory tracking
- The need to separately handle the case when a directory is created (for example, mkdir -p a / b / c). You will receive an event that the directory "a" has been created, but while you hang the handler on this directory, another directory may already be created in it, and you will not receive an event about this.
- The theoretical possibility of integer overflow watch descriptor (wd) as it is given by uint32
FreeBSD, Mac OS X, kqueue
FreeBSD and Mac OS X allow change tracking with kqueue, which is similar in characteristics to inotify and also lacks recursive directory tracking. Also, kqueue takes descriptors as arguments. open files (directories), so when using this API, the restrictions on the number of monitored directories are even stricter.Total:
As you can see, all APIs have their own advantages and disadvantages. The least convenient mechanisms are kqueue and inotify, but they are also the most efficient and reliable. Commercial operating systems provide more convenient mechanisms for tracking changes, but they also have their own peculiarities. I hope you now have a better idea of \u200b\u200bhow hard it is for Dropbox and similar programs that need to get along with all this and carry out reliable and efficient data synchronization :).* Picture taken from
Introduction
Note. Filemon and Regmon are replaced by the same Process Monitor in Windows versions starting with Windows 2000 SP4, Windows XP SP2, Windows Server 2003 SP1 and Windows Vista... However, Filemon and Regmon were retained to support legacy operating systems, including Windows 9x.
Program FileMon monitors and displays in real time all operations with the file system. Its rich capabilities make it a powerful tool for monitoring how Windows is running, tracking how applications are using files, and dLLs, as well as solve problems related to the configuration of system files or applications. Filemon accurately records the time it takes to open, read, write, or delete a file or directory. The status column displays the result of each of these operations. Program FileMon so easy to use that you will become an expert in minutes. It starts tracking as soon as it starts, and its output can be saved to a file for offline viewing. The program has all the possibilities for searching, and if you think that it displays too much information, just set up one or several filters.
Program FileMon runs on NT 4.0, Windows 2000, Windows XP, Windows XP and Windows Server 2003 64-bit Edition, Windows 2003 Server, Windows 95, Windows 98, and Windows ME.
Installation and use
If you have questions or problems, visit the Sysinternals Filemon Forum.
Just run the program FileMon (filemon.exe). To do this, you must have administrator rights. When the program is launched for the first time, it keeps track of all local hard drives... You can use menu commands, keyboard shortcuts, or buttons to clear the program window, select or delete volumes for monitoring, including network volumes (Windows NT / 2K / XP), save the retrieved data to a file, and filter or search the output. toolbars.
If at the last start FileMon any filters have been configured, the next time you start the program, you will be asked to confirm their use. To start it FileMon without asking for confirmation, it is enough to indicate in command line the / q option. Once launched, the program automatically starts logging operations with the file system. To start it FileMonIf you turn off the recording of operations, you must specify the / o parameter on the command line.
When events are written to the output, they are assigned sequential numbers. If Filemon's internal buffers are full, which happens when there are too many operations, gaps appear in the number sequence.
Every time you go out FileMon, the program remembers the configured filters, the position of the window, and the width of the output columns.
Filtration
In the "Filter" dialog box, which is opened using a toolbar button or the "Edit | Filter / Highlight" menu command, you can specify what data should be displayed in the list. The wildcard "*" matches an arbitrary string; filters are not case sensitive. The list displays only the data that is defined in the inclusive filter and is not defined in the exclusion filter. Lines in the filter are separated by semicolons, for example: "filemon; temp". Note for Windows NT and 2000. Since the input and output for file system are performed asynchronously, filtering by the result field cannot be performed.
For example, if the inclusive filter "c: \\ temp" and the exclusion filter "c: \\ temp \\ subdirectory" are configured, the program will track accesses to all files and directories in the c: \\ temp folder, except for those located in the c: \\ temp \\ subdirectory.
Wildcards can be used to specify complex patterns for matching, which allows, for example, to track accesses to specific files from the side of specific applications. So, if the "Winword * Windows" include filter is configured, the program FileMon will show only application hits Microsoft Word to files and directories with the word "Windows" in their names.
With a highlight filter, you can specify which rows in the list should be highlighted. The colors for the highlight are set using the menu item "Edit | Highlight Colors" (Edit | Highlight colors).
Additional filtering options let you select or exclude read, write, or open operations. For example, when working on troubleshooting, often only the operations of opening files or directories are of interest.
Volume selection (Windows NT / 2K / XP / 2K3)
Use the Volumes menu to select or exclude volumes to monitor. You can use the Network item on this menu to monitor access to any network resources, including remote shared folders and UNC path access to remote volumes.
Limiting the amount of output
The "History Depth" dialog box, which can be opened using the corresponding button on the toolbar or the menu item "Edit | History", allows you to specify the maximum number of lines in the output window. A value of 0 means no restrictions.
Search by output
Search the required lines in the output window, you can use the "Find" menu command or the corresponding button on the toolbar. You can repeat the search in the forward direction using the F3 key, and in the opposite direction using the Shift + F3 keyboard shortcut. To start your search from a specific line of output, you must select it by clicking the leftmost margin of that line (ordinal). If no line is selected, new search starts with the first (top-down search) or last (bottom-up search) record.
Options
Program FileMon can show either the execution time of operations or their duration. The "Options" menu and the clock button on the toolbar allow you to switch from one mode to another. Depending on the current mode, a clock or stopwatch icon is displayed on the toolbar. If the mode is set to show the duration of the operation, the value in the Time field corresponds to the number of seconds it took the file system to service a particular request. The menu command "Options | Show Milliseconds" (Options | Show milliseconds) allows you to display the time with millisecond precision.
To window FileMon was always displayed on top of other windows, just select the menu command "Options | Always On Top" (Options | On top of other windows). In addition, using the menu command "Options | Auto Scrol" or the corresponding button on the toolbar, you can specify that the list in the window FileMon did not scroll.
Named pipes and message pipes
email
Since version 4.1, the program FileMon can track file system operations with named pipes and e-mail pipes for Windows NT / 2K Named pipes are commonly used as a communication mechanism in basic NT / Win2K subsystems such as the Local Security Authority Subsystem (LSASS). DCOM also uses named pipes. In addition, networking components such as the browser service work with them. To see operations on named pipes with FileMon, just select the "Named Pipes" item in the "Drives" menu, and then perform some operation on the common network resource or open an application such as Regedt32 that interacts with the security engine.
How FileMon Works
For windows drivers 9x at the heart of the work FileMon the driver lies virtual device Filevxd.vxd. It is loaded dynamically, and during its initialization it installs a file system filter using the VxD service. IFSMGR_InstallFileSystemApiHook, which allows it to be embedded in the call chain of all requests to the file system. In Windows NT at the heart of the work FileMon lies the file system driver that creates filter device objects and binds them to the target file system device objects, which allows the program to view all IRP and FastIO requests sent to disks. When the program FileMon logs a request to open, create, or close a file or directory, it updates an internal hash table, which is a list of correspondences between internal file descriptors and the paths to these files. If a call is made based on a descriptor, the program looks in the table for an entry for that descriptor in order to display the corresponding full path to the file. If the file referenced by the handle was opened before starting FileMon, the corresponding entry in the hash table will not be found, and then the program will simply print the value of the descriptor.
Information about access to the file system is written in ASCII format to a buffer, which is periodically copied to the list displayed in the program window.
additional information
Listed below additional sources information about file windows system 9x:
Windows 95/98 DDK
Mark Russinovich and Bryce Cogswell, "Exploring the Windows 95 Layered File System", Dr. Dobb's Journal, December 1995;
Walter Oney, " System Programming for Windows 95», Microsoft Press, 1996 (every VxD programmer should read this book);
Stan Mitchell, " Internal organization Windows 95 file system», O" Reilly and Associates, 1996.
Below is a list of sources of information about the Windows NT / 2000 file system and / or program FileMon:
David Solomon and Mark Russinovich, “ Internal windows device 2000, third edition", 2000;
Mark Russinovich and Bryce Cogswell, "Exploring the Windows NT File System", Dr. Dobb's Journal, February 1997;
"Service windows programs NT: Inside Look, Windows NT Magazine, February 1999;
Rajeev Nagar, " Internal structure of the Windows NT file system», O" Reilly and Associates, 1997.
Filemon Articles in the Microsoft Knowledge Base
The following articles from the Microsoft Knowledge Base mention using Filemon to diagnose or resolve various errors:- 830903: When you try to open a file on a network folder, it turns out to be read-only or you receive a message stating that the file type cannot be recognized
- 319844: ACC2002 Error Message: ActiveX Control Cannot Create Object
- 830761: Guidelines for Deploying Visual Basic 6.0 Applications
- 837932: Inbound replication of the Active Directory directory service in Windows 2000 Server and Windows Server 2003 logs events 2108 and 1084
- 819612: FIX: Poor performance when you remove the Browser Capability Calculator from the cache
If you are not the only user of your computer and you need to find out what changes have occurred to your files and folders over a certain period of time, you can use special programs that will help you track file changes and get accurate information about all changes to the Windows file system.
How to track file and folder changes?
In this article, I will introduce you to two programs. With FolderChangesView and with with the Disk utility Pulse. Both programs are free. FolderChangeView is completely free. Disk Plus has a free and paid version (few differences)
FolderChangesView: Track folder and file changes
FolderChangesView - small free utility to track all changes that occur to files in a specific folder and section hard disk.
The utility scans in real time a predefined folder or group of folders and displays detailed information about all the changes. The entire result is presented in a convenient table.
First you need to download the program. You can download FolderChangesView for free from the developer's site by. There you can also download a crack, which must be unzipped and dropped into the program folder.
FolderChangesView Customization
The program does not require installation. After starting the program, the settings window will appear.
FolderChangesView Tracking File Changes
In the settings window, you must specify a folder, several folders, or partition hard disk that you need to monitor. In this case, I selected the spysoftnet folder and ticked Scan subdirectories also
In the second line of the settings window, you can set the folders that you do not want the program to track. For example, I do not want the application to track changes in the - tmp folder. After you have installed the folder, be sure to check the Exclude the folowing folders checkbox.
Also, you can set the minimum and maximum size file. After all the settings are selected, click on OK
Disk Pulse: File and Folder Monitor
DiskPulse is another program for monitoring files, folders and hard driveswhich can show changes in the Windows file system in real time.
DiskPulse: How to Track File and Folder Changes The program has a very interesting, in my opinion, possibility of sending notifications by e-mail or executing user commands (actions) in case of detecting all kinds of dangerous critical changes in the system.
If you leave all the default settings as they are, you will receive information about all changes to the entire system. But if you dig into the options a little, you can reduce the amount of data. In the picture below you can see how you can easily do it with a few clicks.
DiskPulse: Select Folder or Hard Drive IN latest version added a diagram showing the number of files, their types and other useful information.
DiskPulse: File and Folder Monitor Download DiskPulse for free from the developer's site you can by.
Let's summarize. Both programs work great and cope with the basic function of monitoring files and folders and are worthy of attention. Program " FolderChangesView»I liked the program more. Although the second program is more functional. The software is free and virus free, so try both and decide which one to use.
These utilities will help you track user activity, as well as identify and analyze the hidden work of malicious programs. About what malware exist we wrote in the article -.
This review did not include 2 more programs for monitoring files and folders. Which I will review in the next article. Therefore, in order not to miss updates, subscribe to our public VKontakte and other social networks.
Video: Disk overview Pulse
These developers generally offer a ton of great free apps, and most importantly, do not overload the installer with tons of useless additional panels and programs "on load", as other companies like to do.
What is FolderChangesView?
FCV is a small portable utility from Nirsoft Labs that allows you to monitor the contents of folders or the entire hard drive for changes. The program issues full list created, modified and deleted files for the entire observation period. You can follow not only local drives, but also behind the network (if you have permission to read them).
You can download FolderChangesView from. The link to download the ZIP-archive with the executable file is at the bottom of the page. After downloading the program, create a folder "FCV" (or with any other suitable name) on your desktop and extract the contents of the archive into it.
Launching and Using FolderChangesView
Since FCV is a portable utility, there is no need to install it. Just open the folder where you unpacked the archive and double click on the "FolderChangesView.exe" file. You may need to confirm the launch of the application in the User Account Control window.
Once launched, select which folder you want to monitor and click "OK". After that, the utility will start monitoring the folder and all its subfolders in accordance with the specified parameters. For example, we decided to watch the Downloads folder. To do this, we indicated its address, as shown in the screenshot below.
After selecting the folder, you can change the monitoring parameters shown in the screenshot above. We decided not to change anything, but you can later experiment with the settings yourself. And don't worry: this won't change your operating system settings, so you don't have to worry about messing up.
Alteration
So, monitoring is configured, now you can change a couple of files and see how this will be reflected in the application. For example, we deleted one file from the Downloads folder, created a new subfolder, and moved the existing files into it.
Here's what FCV tells us about these changes:
The "Paragon Partition Ma ..." file has been deleted.
New Folder was created and changed (change is renaming a folder).
The file "VirtualBox-4.3.20-9699 ..." has been deleted and created. In other words, it has been moved — that is, removed from the original Downloads folder and added to the New Folder.
The file "Oracle_VM_VirtualBox ..." was deleted and created - also due to being moved to a different folder.
File information
The utility shows not only information about operations, but also comprehensive information about the changed files in the folder (all columns can be viewed by scrolling the monitoring window to the right). Here's what the app reports:
1. Filename.
2. The number of changes (Modified Count).
3. The number of creations (Created Count).
4. The number of deletions (Deleted Count).
5. Full path to the file (Full Path).
6. Extension of the modified file (Extension).
7. File Owner.
8. First Time Event.
9. Last Time Event. These two columns come in handy when you need to know how much time was spent, for example, editing a Word document.
10. File Size.
11. Modified Time. This column is slightly different from # 8 and # 9: it shows the time when various attributes of the file were renamed or changed, not when it was opened and saved.
12. Created Time.
13. Attributes.
For convenience, you can click on any file right click mouse and select "Properties" - then all this information can be viewed in one window. Just do not confuse it with the item “File Properties”.
| Materials |
A small program that does not require installation, designed to monitor files in the folder you specified. It allows you to monitor both local and network folder and display notifications both on the Desktop, and via the network or by E-mail! Also, the program has the ability to quickly respond to changes by launching bat files or specific applications.
It just so happened that today everyone, one way or another, depends on the computer. I'm not talking about computer addiction as a disease, no :). It's just that each user in any case stores his personal files on his PC ...
And since several people can work at one computer, then, naturally, you may want to make sure that your data is not going anywhere.
The most radical way is to encrypt your personal folder with a password. However, this is not always convenient, for example, if you need to have access to it over the network ...
In this case, you can seek help special programs to monitor changes to files in the directories you need. They allow you to constantly monitor the selected folders and, if their content changes, notify the user about them. One of the few such free software is an Simple observer.
Comparison with paid analogue
A simple observer, despite the uncomplicated name, is a fairly powerful file system monitoring tool that allows you to monitor both a single folder and an entire disk or even a shared network directory. You can compare it with another paid domestic development - Folder Watchdog Service:
The only drawback of the Simple Observer is the lack of the ability to monitor multiple folders at once. Everything else is only in the black :).
Preparing to work with the program
Another plus of the app is that it doesn't require installation! To start working with the program, you only need to extract it from the archive and place it in any convenient folder (albeit in the one that should be monitored). That's it - we're ready to go :).
Oh yes! I almost forgot to warn you that libraries must be installed on your PC .NET Framework 3.5 (they are usually installed with the system, but they may not be available, so just in case, here is a download link: https://www.microsoft.com/en-ru/download/details.aspx?id\u003d21).
After starting the program, we will see the following:
Don't be alarmed :). Nobody demands money from us - registration is completely free. And if you do not want to disclose your incognito, then you can not register at all! To do this, just click the "Do not register" button.
However, in the future you will have to do this every time you start the program, so it will be easier to enter the required data in the appropriate fields and click "Register".
After that, the main program window will appear in front of us:
It consists of several functional buttons, a link to the watched folder and a status bar, which displays information about the application's activity.
Program settings
Before starting to monitor a folder, I advise you to familiarize yourself with the settings of the Simple Watcher. To get to them, you need to click the "Settings" button:
The program settings are collected in four tabs. And the first of them is "Monitoring". Here are the settings that are responsible for the actual tracking functions. It is worth paying attention to the following parameters:
The second tab is "Alert":
Here we can configure the type of notification about changes in the folder. By default, the notification appears above the tray and is accompanied by a standard signal. However, you can turn off the notification altogether, change the alarm, or set up a local network notification.
If you decide to configure network notifications, be sure to read the information on the link "Important notes on the function".
The Actions tab contains all the functions available for execution by the program after detecting changes in the monitored folder:
Here we can configure:
- launching a specific program or batch script;
- sending a report to an email box;
- sending modified files to an FTP server;
- copy changed files to another folder.
We should especially dwell on the point of work with E-mail notifications. If you want to receive reports (as well as modified files) to your email mailbox, then you need to activate the checkbox "Sending reports by e-mail", and then click the activated button "Settings":
In the settings window, we need to specify the following parameters:
- "SMTP Server". If you use online mail services, the server address usually corresponds to the address of the service itself, preceded by the prefix "Smtp." (eg, smtp.mail.ru, smtp.yandex.ru etc.). If this scheme fails, then the exact address look at your provider's help page.
- "Mailing address". Here you specify the address from which the report will be sent. In order to avoid misunderstandings, it is better to indicate your own E-mail.
- Authentication required checkbox. If your mail server to send letters, authorization is required (usually on all online services), then you need to activate this option and specify the username and password for your mailbox.
- Recipients field. Here, separated by semicolons, we enter a list of addresses to which reports will be sent.
Additionally, you can pay attention to the "Attach new files" option. It allows, together with the notification, to send you files that have changed by mail.
In order not to overload the Internet channel and not waste traffic in vain (if it is paid), you can limit the size of the sent files using the option "Do not send files larger than ...". Below it is also possible, on the contrary, to specify the files that will always be sent. Such a file can be, for example, a program operation log.
At the end of the setting, you can check its correctness by clicking the "Test" button:
If everything is configured correctly, you will receive an email with a test message.
The last tab is "Program":
Collected here are options related to general settings Simple observer, such as autostart at system boot, automatic start monitoring a folder, hiding an application to the tray, etc. Here you can also check for updates and activate the display of pop-up messages (if they are not visible by default).
Observing a folder in the Simple Observer program
Now that you have everything set up as needed, you can start working directly with the Simple Observer program. And first of all, you need to specify the folder for monitoring. By default, the entire C: drive is selected, therefore, to change the path, you need to click the "Folder" button, and in the Explorer window specify the required directory:
I created a separate test folder on the Desktop, but the location of the folder does not matter - the program works equally well with all drives. It is worth mentioning, however, that only the local directory can be specified through Explorer. To monitor remote shared folder its address must be set in the first settings window!
So, we have everything ready - you can start tracking. To do this, press the "Start" button:
After pressing it, the “Stop” button is activated, the “Folder” button is deactivated, and information about the time of the folder observation, the number of changes in it and the time of the last change appears in the status bar. In order to hide the program window, it is enough to minimize it, and it is minimized to the tray. You can call the window again with a single click on the tray icon.
To test the functionality of the Simple Observer, let's put a file in our test folder:
As you can see, the program detected the changes and indicated to us that the folder appeared new file with a specific name. Also, a report on this was sent to my e-mail. Another feature of the Simple Observer message is that if you click on the notification window, your folder will open in front of you, in which the changed file will be highlighted!
Viewing logs
As soon as the Simple Observer detects the first activity in the watched folder, it will automatically create a file watch.log... it text file, which records and stores reports of all events that have occurred.
This file can be opened with a regular Notepad, but it is more convenient to view it with the built-in means of the program. To do this, just click the "Log" button:
Here we will see the event data (action, date, time, file and folder name that have changed) and can quickly find desired entry thanks to the filter system. There is also a button that allows you to quickly clear the log file if it is too "swollen" in size :).
By the way, knowing the name and location of the log file, you can configure the Simple Observer so that it sends a log file along with each message to your email (see the section on settings).
Advantages and disadvantages of the program
- no installation required;
- small size of the utility;
- the ability to track subfolders;
- sending reports to e-mail;
- sending notifications over the local network.
- works with only one folder;
- consumes quite a lot of resources on old PCs without preliminary configuration;
- there is no way to completely hide the work of the program by "regular" means.
conclusions
Today Simple Observer is perhaps the only completely free (even for commercial use) program of its kind!
Therefore, if you decide by all means to install surveillance of personal data on your PC in your absence, then you simply will not find another option :). Moreover, there are practically no drawbacks in the program, except for monitoring only one folder.
P.S. It is allowed to freely copy and quote this article, provided that an open active link to the source is indicated and the authorship of Ruslan Tertyshny is preserved.
