{"id":272,"date":"2021-09-30T15:39:21","date_gmt":"2021-09-30T14:39:21","guid":{"rendered":"https:\/\/www.uptheretheylove.com\/blog\/?p=272"},"modified":"2021-09-30T15:39:21","modified_gmt":"2021-09-30T14:39:21","slug":"reliable-builds-with-a-wobbly-tech-stack","status":"publish","type":"post","link":"https:\/\/uptheretheylove.com\/blog\/2021\/09\/30\/reliable-builds-with-a-wobbly-tech-stack\/","title":{"rendered":"Reliable builds with a wobbly tech stack"},"content":{"rendered":"\n<figure class=\"wp-block-image size-large\"><a href=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/TechStackTopple.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"575\" src=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/TechStackTopple-1024x575.png\" alt=\"\" class=\"wp-image-295\" srcset=\"https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/TechStackTopple-1024x575.png 1024w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/TechStackTopple-300x168.png 300w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/TechStackTopple-768x431.png 768w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/TechStackTopple-1536x863.png 1536w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/TechStackTopple.png 1934w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><\/figure>\n\n\n\n<p>If you&#8217;ve ever worked on a large software project, you know that keeping track of your tech stack can be a full-time job. Especially when your project uses multiple languages and frameworks, it can be a painful process to set up the environment correctly.<\/p>\n\n\n\n<p>In this article, I will explain how I use <a href=\"https:\/\/pypi.org\/project\/uttl.buildout\/\">Buildout<\/a>, an open-source automation tool written in Python, to create reliable builds for my game written in C++, JavaScript, and Python.<\/p>\n\n\n\n<!--more-->\n\n\n\n\n\n<p>Buildout is great because you can put the steps to build your project directly into code. These steps are usually only written down in some internal wiki that has never been updated. Or worse, the information is passed by oral tradition, and the person who knows the exact steps to get the build running left the company three years ago&#8230;<\/p>\n\n\n\n<p>Even as a solo game developer, I often simply <em>forget <\/em>the exact steps needed to build my game. Recording them as a Buildout configuration is a huge relief when I come back to my project after not being able to work on it for a while.<\/p>\n\n\n\n<p>Besides talking about my quest for a stable build process, I also want to introduce <a href=\"https:\/\/pypi.org\/project\/uttl.buildout\/\">uttl.buildout<\/a>, an open-source package for Buildout that will help you smooth out the build process for your own game.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Bootstrapping the process<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large is-resized\"><a href=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/Screenshot142.png\"><img loading=\"lazy\" decoding=\"async\" src=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/Screenshot142-1024x600.png\" alt=\"\" class=\"wp-image-296\" width=\"610\" height=\"357\" srcset=\"https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/Screenshot142-1024x600.png 1024w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/Screenshot142-300x176.png 300w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/Screenshot142-768x450.png 768w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/Screenshot142-1536x900.png 1536w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/Screenshot142.png 1922w\" sizes=\"auto, (max-width: 610px) 100vw, 610px\" \/><\/a><figcaption>I&#8217;m calling this an &#8220;early build&#8221; of my game because I don&#8217;t want you to criticize my graphics. I can do that perfectly well enough on my own.<\/figcaption><\/figure>\n\n\n\n<p>The first commit for <em>Up There They Love<\/em> was on September 13th, 2017, and it contained a Visual Studio project, a <code>Main.cpp<\/code>, and <code>bootstrap.py<\/code>. All the bootstrap script did in that first version was to copy over some required Qt libraries to the build directory.<\/p>\n\n\n\n<p>The main goal for the script was to always get a working game by running <code>python bootstrap.py<\/code> in a console window. I knew this could become very complicated over time, so I wanted to have it written down in code straight away. Over time, I expanded the script to find tools like Visual Studio and CMake, build dependencies automatically, and package the game for release.<\/p>\n\n\n\n<p>My tech stack is a combination of Qt, SDL, Chromium, custom JavaScript, Sass, Grunt, and Webpack. It&#8217;s quite difficult to keep track of these different technologies, I have to be mindful of switching between writing C++ and JavaScript, for example. I don&#8217;t want to then also waste my time trying to remember to copy certain files over if I want to actually run the build.<\/p>\n\n\n\n<p>(Yes, I use both Webpack and Grunt. No, I don&#8217;t want to talk about it.)<\/p>\n\n\n\n<p>That&#8217;s why it was so important to me to start with this bootstrap script: it helps to keep things running smoothly. When I make time to work on my game, I can run <code>git pull &amp;&amp; python bootstrap.py<\/code> to get in the right headspace straight away.<\/p>\n\n\n\n<p>If something isn&#8217;t working with the script, I fix it immediately. I don&#8217;t want to be in a situation where the script cannot reliably create a build. Because that effectively means I won&#8217;t remember how to build my game.<\/p>\n\n\n\n<p>But after three years of intermittent development, this script had grown to around 2400 lines of Python. I still considered this to be a manageable size, but I was starting to experience some growing pains.<\/p>\n\n\n\n<p>So I started to look for something better.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Breaking it down<\/h2>\n\n\n\n<p>Ultimately, my bootstrap script solves some very practical problems:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Find required tools on the system (Visual Studio, git, etc.)<\/li><li>Resolve third-party dependencies (Qt, Grantlee, Chromium, etc.)<\/li><li>Compile and link everything<\/li><li>Package it up<\/li><li>Look cool while doing it<\/li><\/ul>\n\n\n\n<p>This is easy enough to solve with some Python scripting. For example, I use the registry to check if CMake is installed:<\/p>\n\n\n\n<pre title=\"\" class=\"wp-block-code\"><code lang=\"python\" class=\"language-python\">def ToolsCMake():\n\tprint('CMake:')\n\n\twith PrintScopeLock() as lock:\n\t\ttry:\n\t\t\tkey = winreg.OpenKey(winreg.HKEY_CURRENT_USER, r\"SOFTWARE\\Kitware\\CMake\", 0, winreg.KEY_READ)\n\t\t\tinstalled = winreg.QueryValueEx(key, 'installed')\n\t\texcept OSError:\n\t\t\tprint('CMake is not installed.')\n\t\t\tprint('')\n\n\t\t\treturn False\n\t\t\n\t\tif installed[0] != 1:\n\t\t\tprint('CMake is not installed.')\n\t\t\tprint('')\n\n\t\t\treturn False\n\n\t\tversion = None\n\n\t\tp = subprocess.Popen(['cmake', '--version'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n\t\tfor line in iter(p.stdout.readline, b''):\n\t\t\tmatch = re.match('.*version ([0-9]+\\\\.[0-9]+\\\\.[0-9]+)', str(line))\n\t\t\tif match:\n\t\t\t\tversion = match.group(1)\n\t\t\t\tbreak\n\n\t\tif not version:\n\t\t\tprint('Failed to determine CMake version.')\n\t\t\tprint('')\n\n\t\t\treturn False\n\n\t\tprint('INSTALLED ' + version)\n\t\tprint('')\n\n\t\treturn True<\/code><\/pre>\n\n\n\n<p>But what I ran into is that the script would run very inefficiently. It would do things like building all the dependencies every time. I didn&#8217;t have an easy way to check if those steps were necessary.<\/p>\n\n\n\n<p>It&#8217;s not like this is a novel problem, though. For example, MSBuild also has to track changes on files and only run steps when something has changed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Intermediately awesome<\/h2>\n\n\n\n<p>A perfectly well-adjusted developer like yourself has likely never taken a look at the output in the &#8220;intermediate&#8221; folder for your projects. The compiler puts the binary blobs for the linking process there. Nothing interesting for us humans.<\/p>\n\n\n\n<p>And that&#8217;s mostly correct, but this folder also contains &#8220;.tlog&#8221; files. These files contain instructions for the compiler on how to build your source files and are actually perfectly human-readable.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><a href=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/image-7.png\"><img loading=\"lazy\" decoding=\"async\" width=\"595\" height=\"515\" src=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/image-7.png\" alt=\"\" class=\"wp-image-292\" srcset=\"https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-7.png 595w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-7-300x260.png 300w\" sizes=\"auto, (max-width: 595px) 100vw, 595px\" \/><\/a><\/figure>\n\n\n\n<p>For example, the &#8220;CL.command.1.tlog&#8221; file is used for the cl.exe tool, i.e. the C++ compiler that comes with Visual Studio. It may look daunting, but it&#8217;s actually quite straightforward. All it contains are the paths to the input files and the flags required to compile them:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">^C:\\PROJECTS\\SSSG\\SOURCE\\SERVER\\RANDOMNESS.CPP\n\/c \/I\"C:\\PROJECTS\\SSSG\\PARTS\\CHROMIUM-EMBEDDED\\\\\" \/I\"C:\\PROJECTS\\SSSG\\PARTS\\GRANTLEE-MASTER\\INCLUDE\" \/I\"C:\\PROJECTS\\SSSG\\PARTS\\QHTTP-MASTER\\SRC\\\\\" \/IC:\\PROJECTS\\SSSG\\DEPENDENCIES\\QTFORWARD\\ \/IC:\\QT\\5.15.2\\MSVC2015_64\\INCLUDE\\ \/IC:\\PROJECTS\\SSSG\\GENERATED\\ \/IC:\\PROJECTS\\SSSG\\SOURCE\\SERVER\\ \/Zi \/nologo \/W3 \/WX- \/diagnostics:classic \/Ox \/Ob2 \/Oi \/Ot \/Oy \/GT \/GL \/D QT_NO_DEBUG \/D QT_DLL \/D QT_CORE_LIB \/D QT_GUI_LIB \/D QT_SQL_LIB \/D QT_NETWORK_LIB \/D NDEBUG \/D _SECURE_SCL=0 \/D WIN32 \/D _WIN32 \/D _WINDOWS \/D _CRT_SECURE_NO_DEPRECATE \/D _MBCS \/GF \/Gm- \/EHsc \/MD \/GS- \/Gy \/fp:fast \/Zc:wchar_t- \/Zc:forScope \/Zc:inline \/Zc:rvalueCast \/GR- \/Fo\"C:\\PROJECTS\\SSSG\\INTERMEDIATE\\SSSG\\X64RELEASE\\\\\" \/Fd\"C:\\PROJECTS\\SSSG\\BUILD\\SSSGRELEASE.PDB\" \/Gd \/TP \/FC C:\\PROJECTS\\SSSG\\SOURCE\\SERVER\\RANDOMNESS.CPP<\/code><\/pre>\n\n\n\n<p>The first line is always the input file as an absolute path and written in uppercase. This is to ensure no confusion about a source file and the main reason why you should never check these files into source control!<\/p>\n\n\n\n<p>The second line lists all of the compiler flags. These come from both the Visual Studio project and from the settings specific to each file. What&#8217;s very important to know is that these flags are always in the same order.<\/p>\n\n\n\n<p>With the information in this intermediate file, MSBuild is able to figure out when it needs to compile a source file. If a destination or intermediate file is missing, that obviously means it needs to build the source. But similarly,  MSBuild can compare the compiler flags against what it has recorded. If the flags were changed, the source file needs to be rebuilt.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Buildout<\/h2>\n\n\n\n<p>All of this was a very roundabout way to say that buildout works in exactly the same way, but you don&#8217;t have to use it for compiling C++. Buildout automates <em>anything you want<\/em>, including MSBuild itself!<\/p>\n\n\n\n<p>When you run <code>buildout<\/code> on the command line, it looks for a <code>buildout.cfg<\/code> file by default. This is an INI configuration file where you subdivide the work into sections called &#8220;parts&#8221;:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">[buildout]\nparts =\n    server-build-dependencies\n    server-build-game\n\n[server-build-dependencies]\nrecipe = uttl.buildout:devenv\nexecutable = ${devenv:path}\nsolution = ${buildout:directory}\/SSSG_Dependencies.sln\nbuild = Release\nproject = InkWrapper\n\n[server-build-game]\nrecipe = uttl.buildout:devenv\nexecutable = ${devenv:path}\nsolution = ${buildout:directory}\/SSSG.sln\nproject = SSSG\nbuild = Release\nexe-path = ${buildout:directory}\/build\/SSSG${:build}.exe\nalways-install = 1<\/code><\/pre>\n\n\n\n<p>Each part is &#8220;installed&#8221; with the specified options, which means the associated recipe (script) is called by Buildout.<\/p>\n\n\n\n<p>And of course, these settings resolve to&#8230; an intermediate file!<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">[server-build-dependencies]\n__buildout_installed__ = C:\\SSSG\\parts\\ink-package\\ink-engine-runtime\\bin\\Release\\netstandard2.0\\ink-engine-runtime.dll\n\tC:\\SSSG\\parts\\ink-package\\compiler\\bin\\Release\\netstandard2.0\\ink_compiler.dll\n\tC:\\SSSG\\dependencies\\InkWrapper\\bin\\Release\\InkWrapper.dll\n__buildout_signature__ = uttl.buildout-1.2.4-py3.9.egg zc.buildout-0aec1332ee57ae0a5a956ad9fcd74357 setuptools-0aec1332ee57ae0a5a956ad9fcd74357\nargs = C:\\SSSG\\SSSG_Dependencies.sln \/Build Release \/Project InkWrapper\nbuild = Release\nexecutable = C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\Common7\\IDE\\devenv.com\nproject = InkWrapper\nrecipe = uttl.buildout:devenv\nsolution = C:\\SSSG\/SSSG_Dependencies.sln\nsolution-path = C:\\SSSG\/SSSG_Dependencies.sln\nworking-dir = C:\\SSSG<\/code><\/pre>\n\n\n\n<p>Buildout uses the generated <code>.installed.cfg<\/code> file to track changes between runs. When the settings differ from what was seen before, the recipe is called again.<\/p>\n\n\n\n<p>Understanding how this system worked gave me the confidence to start converting my custom-written Python script to a configuration file for Buildout.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Something found lacking<\/h2>\n\n\n\n<p>With the building blocks that Buildout provides, I&#8217;m able to run the 67 (!) configuration steps required to build my game. That&#8217;s actually a lot more steps than were in my original script because each step is a lot more fine-grained.<\/p>\n\n\n\n<p>For example, instead of one giant Python function that takes care of building and linking Chromium, I now have seven steps in Buildout to download the package, build the different versions, and deploy files to the build directory:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">[buildout]\nparts +=\n\tchromium-package\n\tchromium-build-libcef-debug\n\tchromium-build-libcef-release\n\tchromium-deploy-resources\n\tchromium-deploy-locales\n\tchromium-deploy-shaders\n\tchromium-deploy-dlls<\/code><\/pre>\n\n\n\n<p>Crucially, this means that each step is run only when it was invalidated. In the case of Chromium, that means the library is only rebuilt when I change the version of the package that should be downloaded.<\/p>\n\n\n\n<p>Splitting the steps saves a lot of time compared to what the old script did, which was to build the library and copy files to the game&#8217;s build folder regardless of whether they were out of date.<\/p>\n\n\n\n<p>Don&#8217;t get me wrong, switching from a custom Python script to a Buildout configuration wasn&#8217;t easy. While the tool is incredibly flexible and boasts a vibrant community of package and extension writers, I had some pretty niche problems to solve. For example, I couldn&#8217;t find an existing recipe for executing CMake commands, which I needed to build some of my dependencies.<\/p>\n\n\n\n<p>Here are a few Buildout packages I can definitely recommend:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li><a href=\"https:\/\/pypi.org\/project\/hexagonit.recipe.download\/\">hexagonit.recipe.download<\/a> &#8211; Download and unpack zip files<\/li><li><a href=\"https:\/\/pypi.org\/project\/slapos.recipe.build\/\">slapos.recipe.build<\/a> &#8211; Fetch repositories from source control<\/li><li><a href=\"https:\/\/pypi.org\/project\/mr.scripty\/\">mr.scripty<\/a> &#8211; Ad hoc Python scripting<\/li><\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Scratching an itch<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><a href=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/image-8.png\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"303\" src=\"http:\/\/178.104.38.113\/wp-content\/uploads\/2021\/09\/image-8-1024x303.png\" alt=\"\" class=\"wp-image-298\" srcset=\"https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-8-1024x303.png 1024w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-8-300x89.png 300w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-8-768x227.png 768w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-8-1536x455.png 1536w, https:\/\/uptheretheylove.com\/blog\/wp-content\/uploads\/2021\/09\/image-8.png 1888w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/a><figcaption>Not only is PyPi a great word to say, they even host cool packages like uttl.buildout!<\/figcaption><\/figure>\n\n\n\n<p>To both solve my own problems and give back to the Buildout community, I&#8217;ve written a package called <a href=\"https:\/\/pypi.org\/project\/uttl.buildout\/\">uttl.buildout<\/a>. It focuses on providing tools for developing games on Windows, so it comes with the following recipes:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>uttl.buildout.command &#8211; Run an executable with arguments<\/li><li>uttl.buildout.copyfile &#8211; Copy files between directories<\/li><li>uttl.buildout.versioncheck &#8211; Get a versioned executable<\/li><li>uttl.buildout.cmake &#8211; Run CMake commands<\/li><li>uttl.buildout.qtdeploy &#8211; Deploy&nbsp;Qt&nbsp;libraries<\/li><li>uttl.buildout.qmake &#8211; Run&nbsp;QMake&nbsp;commands<\/li><li>uttl.buildout.devenv&nbsp;&#8211;&nbsp;Build&nbsp;projects&nbsp;with&nbsp;Visual&nbsp;Studio<\/li><li>uttl.buildout.dotnet-restore &#8211; Restore&nbsp;.NET&nbsp;packages&nbsp;using&nbsp;NuGet<\/li><li>uttl.buildout.inklecate &#8211; Compile&nbsp;.ink&nbsp;files&nbsp;to&nbsp;JSON<\/li><\/ul>\n\n\n\n<p>You can find more detailed documentation for each recipe on <a href=\"https:\/\/github.com\/MrHands\/uttl-buildout\">GitHub<\/a>.<\/p>\n\n\n\n<p>The package is free to use for personal and commercial projects. It comes with an MIT-0 license, so you don&#8217;t even have to give credit if you don&#8217;t want to.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Buildout allows me to consistently run the 67-step process required to build my game. And because it&#8217;s all written down as code, the computer will tell me when something goes wrong.<\/p>\n\n\n\n<p>You can&#8217;t beat a reliable process like that!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>If you&#8217;ve ever worked on a large software project, you know that keeping track of your tech stack can be a full-time job. Especially when your project uses multiple languages and frameworks, it can be a painful process to set up the environment correctly. In this article, I will explain how I use Buildout, an [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[9,26,30],"class_list":["post-272","post","type-post","status-publish","format-standard","hentry","category-programming","tag-buildout","tag-programming","tag-tech"],"_links":{"self":[{"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/posts\/272","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/comments?post=272"}],"version-history":[{"count":0,"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/posts\/272\/revisions"}],"wp:attachment":[{"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/media?parent=272"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/categories?post=272"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/uptheretheylove.com\/blog\/wp-json\/wp\/v2\/tags?post=272"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}