How to proxy a console app
TL;DR If one console app calls another, under many circumstances child's output may be lost. Proper output redirection that always works requries special effort.
ConsoleProxy binaries
Source code (github repo)
ConsoleProxy.cpp
The Context
Console application git.exe can be invoked interactively from the command line, or in hidden mode from a GUI front-end (TortoiseGit). We need to replace git.exe with a proxy application, that does some lightweight processing, e.g. logs the arguments, and then calls the original git.exe. We cannot recompile neither git.exe, nor the front-end. The caller should see no difference between invoking the proxy and the original git.exe, both in interactive and in hidden mode.
The Problem
It was relatively easy to get the proxy working on the interactive command line. However, it refused to work with TortoiseGit. When git.exe path is set, TortoiseGit calls git --version and expects to see something like git version 2.17.1.windows.2, but with the proxy it was getting an empty string. The same command worked properly on the command line, even with its output redirected to a file.
Specifically, TortoiseGit calls git.exe with DETACHED_PROCESS flag, and redirects its output to a pipe (source code) . This works great for regular git, but not for the proxy. The grandchild process, the original git.exe, does not inherit the parent's redirected standard output, and whatever it prints to stdout gets lost in the ether.
The Solution
ConsoleProxy tool is an example of a transparent logging proxy, that can replace an original console program like git, record command line arguments in a file, and then transparently invoke the original application. Here are some considerations that were important for this project.
Why DETACHED_PROCESS is a bad idea
Ideally, GUI applications should stick to CREATE_NO_WINDOW when inoking hidden console apps, and never use DETACHED_PROCESS. Creating console process in DETACHED_PROCESS mode causes a number of problems:
- Grandchildren's output will be lost.
- Grandchildren will popup new console(s).
- International characters in the output may have unexpected encoding.
How to handle DETACHED_PROCESS in proxy
If the proxy has been created in any of non-detached modes, it can invoke the child process with default flags and no I/O redirect, and everything just works.
If the proxy has been created in DETACHED_PROCESS mode, it must handle the child differently. It cannot just use default flags, as the child will then create a new console, which would flash, and its output would go to that console, and will be lost. Instead, the proxy must explicitly redirect the child's standard output to its own standard output, and run the child in DETACHED_PROCESS mode. This will have the same drawbacks as decscribed above, e.g. output of grandchild processes would be lost, but the same would happen if the child were directly invoked by the parent, so this achieves 100% transparency.
How to detect DETACHED_PROCESS
Unfortunately, Windows does not provide an API to retrieve process creation flags. The proxy can detect DETACHED_PROCESS mode indirectly, by checking whether it ahs a console window (GetConsoleWindow), and whether console output code page is set to zero (GetConsoleOutputCP). If the former returns NULL, and the latter returns 0, we are in detached mode.
Exploratory tools
Trying to understand why TortoiseGit cannot access git output when I use proxy led me to research a number of aspects of console applications, and build exploratory tools. Research results are in the "see also" section below, I had to break them into multiple documents. The tools, briefly, are:
| DescribeOutput | Describes current console and prints information to standard output and to a file. |
| Exec | Console application that executes a command with given process creation flags. |
| ExecW | GUI application that executes a command with given process creation flags. |
| ShowOutput | GUI application that executes a command with given process creation flags, redirects output to a pipe and displays it in a message box. |
More details in readme.md on Github: https://github.com/ikriv/ConsoleProxy.
Why C++
I normally write tools in .NET, since the development time is faster, and .NET class library often makes much more sense than Win32 API. However, I discovered that .NET Process is unable to redirect standard output of a child to an arbitrary stream. It insists on creating its own stream that the parent process is supposed to read. In my case this was useless, I just wanted to redirect the child's standard output to the parent's standard output.
In fact, I have a .NET implementation of something similar to ConsoleProxy that uses P/Invoke, but it was at least as painful as C++, coupled with the lack of header files: every function prototype must be added by hand. So C++ it was. Besides, I enjoy going back to C++ every now and then and rediscovering how super powerful and super ugly it is. For example, I gave up on reading system time with milliseconds precision in a standard-compliant way, I just used GetSystemTime. And string streams... Oh, well. At least I got them working.
See Also
The research results touched multiple subjects that deserved their own articles:
- Exact meaning of console creation flags.
- What is the difference between CREATE_NO_WINDOW and DETACHED_PROCESS?
- Under what circumstances is new console window created?
- Where does the standard output go when not redirected explicitly?
- How international characters are handled in console applications.
- How to escape command line arguments to pass them to the child process.
Feedback
If you have questions or comments, as usual, feel free to
Leave feedback.