Cache Troubleshooting


If you've followed the setup instructions and still can't seem to make it work, this guide is for you.

General DebuggingGet more information about cache operations
Cache KeysCache Keys overview
Source HashThe Source component of Cache Keys
Command Line HashThe Command Line component of Cache Keys
Toolchain HashThe Toolchain component of Cache Keys
Precompiled Header HashThe Precompiled Header component of Cache Keys
Cache VersionThe FASTbuild cache version component of Cache Keys
LightCache ConsiderationsAdditional information regarding behaviour when using the LightCache
General Debugging

Enable -cacheverbose

You can use the -cacheverbose command line option to emit additional information about cache operations, and in particular, the Cache Key generated for each Object File being compiled.

Combining -cacheverbose with -cache, -cacheread, or -cachewrite will output additional information in the following format:

12>Obj: C:\p4\depot\tmp\x64-Debug\Tools\FBuild\FBuildCore\FBuildCore_Unity3.obj - Cache Miss: 35 ms 'E90519265A1719FA_3AC8C650_6C30067E14AB4880-0000000000000000.A' 12>Obj: C:\p4\depot\tmp\x64-Release\TestFramework\TestFramework_Unity1.obj - Cache Store: 157 ms (Store: 150 ms - Compress: 1 ms) (Compressed: 25684 - Uncompressed: 51916) 'BA4252DC6B561582_97907E3A_6C30067E14AB4880-0000000000000000.A' 10>Obj: C:\p4\depot\tmp\x64-Profile\Core\Core_Unity1.obj - Cache Hit: 81 ms (Retrieve: 73 ms - Decompress: 1 ms) (Compressed: 452262 - Uncompressed: 1044600) 'FD9B691EF509B98D_3409CAAE_6C30067E14AB4880-0000000000000000.A'
Cache Keys

The Cache Key is a unique identifier representing an entry in the cache. Cache Keys are generated by combining hashes of various inputs that feed into compilation of an Object File. Cache Keys are in comprised as follows:

/- Command line hash / /- Precompiled header hash / / E90519265A1719FA_3AC8C650_6C30067E14AB4880-0000000000000000.A \ \ \- Cache version \ \- Toolchain hash \- Source hash
Key PartDescription
Source hashHash of source code involved in the compilation of the Object File
Command line hashHash of command line passed to compiler
Toolchain hashHash of compiler binaries as specified by Compiler()
Precompiled header hashHash of precompiled header (if used)
Cache versionFASTBuild's cache version

For an object to be retrieved from the cache, the Cache Key used during storage and retrieval must match exactly. Differences in components of the Cache Key imply compiled output results will differ, inhibiting caching.

If cache hits are expected but not realized, checking for differences between Cache Keys will highlight the component resulting in said cache misses. For each component, you can troubleshoot further using the information below.

Source Hash

The source code used for compilation must precisely match for caching to be possible. i.e. Two machines must be compiling the same source code, resulting in the same preprocessed output.

You can view the preprocessed source yourself by directly invoking the compiler with an appropriate command line. You can first invoke FASTBuild with -showcmds to see what command line is being using:

fbuild.exe [target] -cachewrite -showcmds -clean

You can then invoke the preprocessor command line yourself, piping the output to a file as per the following example:

cl.exe someCpp.cpp someObj.obj /E > preprocessed.ii

If you do this on two different machines, you can compare the results to look for discrepancies.

  • NOTE: If you are using LightCache, you will need to manually append the preprocessor option (usually /E or -E)

Common causes of Source Hash discrepancies are detailed below:

Source Files

Machines must be compiling the same code to benefit from caching and the source files must match. A single byte in a single file can result in different compiler output, and thus can prevent caching. Depending on the compiler, line endings may also matter.

Command Line Defines

The defines provided to the compiler on the command line will alter the generated code. Machines must be using the same compiler defines to benefit from caching. (See Command Line Hash below.)

__DATE__ and __TIME__ Macros

Particular care should be taken to avoid __TIME__ and __DATE__ macros which cause cache mismatches due to their continually changing nature (every second and every day respectively).

These macros are usually best avoided anyway, as they can be misleading since they reflect the time an Object File is generated, rather than when an executable or dll was generated. A post-link "stamping" process should usually be preferred for injection of date/time strings into final build artifacts.

If complete removal of these macros is not possible, limitting their use to only certain Object Files will still allow other Object Files to generate consistent Cache Keys.

Command Line Hash

The command line used to invoke the compiler is hashed. This includes the full path to the files being compiled. As such, the source code and working directory must match between multiple machines.

You can use the -showcmds option to view the command line being used to check for differences as follows:

On one machine, perform a write-only operation:

fbuild.exe [target] -cachewrite -cacheverbose -showcmds -clean

On another machine, perform a read-only operation:

fbuild.exe [target] -cacheread -cacheverbose -showcmds -clean

If you do this on two different machines, you can compare the results to look for discrepancies.

  • NOTE: If you are using LightCache, the command line will not be emitted. You make like to disable the LightCache while troubleshooting.
  • NOTE: The .UseRelativePaths_Experimental feature of Compiler() may adequately relax this limitation on full paths in the future.
Toolchain Hash

The Toolchain Hash represents the compiler binaries and additional files as specified by the Compiler() in your FASTBuild config. As different compilers and compiler versions can emit different results, the toolchain must match for caching to be successful. Ensure that your machines are using the same bff files, specifying the same binaries and that they are identical.

  • NOTE: Versioning of compiler binaries along with source files in source control is recommended.
Precompiled Header Hash

If using Precompiled Headers when compiling an Object File, the hash of the precompiled header associated with an Object File is calculated. When not using Precompiled Headers, this component is always 00000000.

For Precompiled Headers (which can also be cached), all Cache Key components are treated normally and the Precompiled Header component is 00000000.

To troubleshoot discrepancies in the Cache Key for a Precompiled Header itself, use the same steps as for a normal Object File.

Cache Version

The Cache Version is used by FASTBuild to allow for invalidation of existing caches when breaking changes to the caching format are made. These changes are rare. If you have a different Cache Version in your Cache Key, you are using incompatible FASTBuild versions.

LightCache Considerations

The LightCache feature replaces the use of the Compiler's preprocessor with FASTBuild's own internal processor. This accelerates Source Hash generation by an order of magnitude, but may add some additional considerations when troubleshooting.

No Preprocessor

Since there is no preprocessor invocation, there is no preprocessor command line that you can examine with -showcmds. As a work-around, you can disable use of the LightCache until you have caching working.

Different Source Hash

The LightCache uses a different mechanism for hashing the source files and as such the Source Hash component of the Cache Key differs compared to using the Compiler's preprocessor. IF debugging caching, be sure to consistently use (or not) the LightCache to ensure you are comparing comparable Cache Keys.