This site is about programming, hobbies, hobby programming, programming as a hobby, hobbies that require programming, and other hobby and programming related topics. Amongst my interests, there are that I'll focus on this site:

At this point, this page is all there is. No CSS. No styling at all. Just a few HTML tags. I've also added a .editorconfig file to standardize some formatting in Visual Stuido Code and other editors I plan on using. And a ReadMe.md for the project. It's now time to submit, and then get Visual Studio Online set up for auto deployment by FTP in a Continuous Integration loop.

The next step was to get Visual Studio Online to FTP this data up to the site every time I commit a check-in to the repository. This turned out to be fairly easy to do. When you enter the Build page, help text comes up that's a bit consfusing but walks you through the process:

And that's all there is to it. At least, that's how it appears. I'm about to check in, so if you're reading this text then it worked...

Bootstrap and Scripts from CDNs

I've added basic Bootstrap support to start making this look like a decent page. There's now a menu (which does nothing since it links to this page). And I've done an "npm init" to start adding tooling on the development side, but that's invisible to you here. Currently I'm using Bootstrap 4 from a CDN, but I intend to pick up the NPM package and start using Gulp or another tool to build some actual CSS.

The choice of NPM here is an example of the kind of decision that you have to make and revisit constantly in a web development project. Every choice you make ties you in to a number of new tools and restrictions. Each choice is a forcing function that molds the site in subtle ways moving forward. I considered using yarn, the new hotness, mostly becasue I expect that I'll be doing some of the development on this site from my laptop, and yarn stores the packages it downloads into a central store. That makes it more likely that I can get a package when I'm offiline, provided I've used it before. However, looking at documentation for recent updates, this has been added to npm. Since yarn is an npm wrapper, I'm sticking with npm for now.

The key, in my opinion, is to realize when it's time to make a choice and make it. Most choices can be recovered from later in the development cycle if you decide you want to change. Now, there are exteremes to that. If I start with Node.js and end up moving to Ruby On Rails, I'll have a lot of work to do. But knowing that there is a lot of infrastructure around Node.js, while Rails locks me into Rails, helped me choose node.js for this project. And Node.js tied in better with what I want to focus my learning on for this project, especially JavaScript. So recognize when you have a choice, make the choice, and then change it if the apparent benefits outweight the reason you want to change it. Don't wait until you know everything. Pick a place to dive, and dive in.

TypeScript, React and Webpack Installed

At this point in the development cycle, it feels like it's time to start making each update to the web site a separate blog entry. And I have no capability to add blog entries, and only a single page. So it's time to start building some infrastructure to extend the site out.

Based on the technologies that I'm most interested in, which is really often the core of hobby programming, I'm going to use React for my user interface. I'm a fan of the Virtual DOM model, and it's what I'd use for a full scale business site at this point. I'd also consider using something like create-react-app to help with the scaffolding. But that seems counter to the goals of this site, and of building a site from scratch.

I also intend to use TypeScript. There's a lot of JavaScript syntax that feels awkward to me, especially when I have more experience with C# and C++ programming. While ECMAScript is expanding to pick much of TypeScript's capabilities, I've decided to stay with TypeScript. Since I'm building the site in Visual Studio Code, I'll get some nice syntax coloring and code completion as I move forward.

Also, using Visual Studio code, I have no spelling or grammar checking going on in this page. I will need to address that at some point, and perhaps start writing the core text in an editor with those. However, I'll allow myself the possibilities of typos for now, and keep moving forward on the code. There isn't anything driving any traffic to this page, so I'll worry about that kind of polish when there's enough content to warrnt people coming here.

So, we're going to need TypeScript and React. Using TypeScript also means I'll need the @types files that come along with my projects. And since my pages will start to be compiled, I'll likely need WebPack for packaging things up. So time to start installing. I want to keep the devloper tools out of the deployment. That means installing TypeScript and Webpack in npm using the -D flag, which indicates that the packages that are installed don't need to be on the server. All the types are developer packages, while react will be installed as part of the project.

When I did this, I soon discovered that I had hundreds of files showing up as unstaged in the git repo. That's becasue the node_modules directory wasn't excluded from the project. I need a .gitignore file to scope my git changes. Again, in a production project, I'd get a good default from create-react-app, but I'll roll my own and we'll update it as needed moving forward. Right now, let's just exlude the node modules.

And that's a good chunk for a commit. Using git, I believe it's best to commit often, as long as your tests pass. Since we have no tests, all 0 of them passed, so we're good to check in. Once we have some actual code, that will change, but for now we'll stop here. Another benefit of frequent check-ins is that it makes it easier to roll-back to an earlier version if you make a mistake, without losing a lot of work. And it also limits the size of the blog entries, which makes them easier to write if you're going one entry per commit.

Development Servers and Web Hosting

Up to this point, I have been building the site out of a single HTML page. This is an unwieldy way to build a site, and it doesn't make it easy to extend things later on. Nobody is going to want to read a single page that gets much longer than the current page. Also, anybody who does read this site is likely to want to be able to read in reverse-chronological order. If they've read anything on this page, they'll want the new bits. That's why blogs tend to be ordered newest-to-oldest.

That leads me to infer that I should be makeing the blog entires not in a single page, but a single HTML file per blog entry. Typical content management systems like WordPress or Joomla use a database to store the content. But I don't want to deal with schemas and locking myself down to a database migration path this early into the project. So the incremental method, following the YAGNI (You Ain't Gonna Need It) principle, is to break the blogs entires out into individual files and have the main page show the latest first.

That in turn is going to require some scripting to generate the main page. Ideally, we'll have the main page show the most recent couple of posts, a "Next/Previous" interface, and possibly keyword or title search. And since we don't want the client to have permission to randomly traverse the site and generate this on the fly for every client, we need some kind of a server archiecture.

I've decided to use React, the current hotness in web programming. This is mostly due to the fact that React is being adopted where I work, and building a React app in this "first principles" approach is a great way to bring myself up to speed. That in turn helps to drive the selction of server software to use. I'm already going to be writing client side code in TypeScript, so doing the same for the server means one less language to deal with. And that, as often happens with implementation decisions, drives yet another change. A change we're going to have to deal with first before I can move forward.

My current web host doesn't support Node.js hosting on my plan. And given their infrastrucuture I'd have to get non-shared server, an expense I don't want to take. So it's time to move to another host. Thanks to MSDN, I've already got some credit on Microsoft's Azure servers that will allow me to host this site there and have the Node application run on Microsoft's cloud. Then I can adjust the DNS entries on my host and have the site served from there. So let's look into what we need to do to make that happen.

So, where to dig in to all of this? I can't deploy a Node.js app to Azure until I have a Node.js app to deploy. So I need to get the app working locally. We're going to want "npm start" to launch a web app that will serve our single page back to use when we visit the site. Not incredibly exciting, but necessary. So let's look at how we configure npm and the test server to start serving us our application. Again, this would have been much easier if I'd gone with "create-react-app" from the beginning, but this is about being pedantic and documenting the whole process. Just to see what happens, let's try "npm start" now:

Obviously we don't have a script for node to run. We know that we intend to use webpack eventually, so I'm going to eschew the typical Agile advice here and move straight to using webpack-dev-server. There are many development servers to choose between, but I'll pick one that I believe is going to meet my needs later on. Given that webpack is designed to be part of the development cycle, I'll make the logical leap that I want to use it here. And we need to use something, so let's get configuring. Install it:

I haven't configued webpack yet, so I'm going to need to create a webpack.config.js file. I'll spare you all of the errors I had getting this working, and all the web pages I had to peruse to learn this. Just know that we always need to have an "entry" in the webpack.config.js, or webpack-dev-server won't start up. Since it's using webpack, it wants to compile something before it server it. In this case, I'm going to hard-code the entry to our HTML file, which is entirely a stop-gap measure and not what we want to do for real. Now the webpack compile will fail, but I'll be able to see the page.

It isn't pretty, but it gets the job done. What happens when we do an "npm start"?

We fail to compile, but the server still starts up. And since it's routed to our page, we get the correct content on the page. So we have successfully configured webpack-dev-sever to be our development enviornment. We don't have any JavaScript files to display, and we're still using our monolithic HTML file. That's okay, we're one step closer to where we want to be. And that's the nature of iterative processes. Worrying about getting every detail perfect can prevent you from making any forward progress. Once we have sufficient forward progress, we can begin putting tests in place to ensure that our incremental changes don't break anything. But for now, we have expanded the website with a new blog entry and set ourselves up a development server that will serve us well in our next steps of writing an app that backs this site and haveing webpack build that up into a deployable bundle we can send to Azure. We've got a nice list of tasks moving forward. So it's time to complete this one. Let's format the blog entry, get it submitted, push it to Visual Studio Online, and update the current site.

Incremantal webpack Improvements

I'm still not sure exactly how I'm going to lay out the built webpack bits, or exactly how I'm going to tie React into this project. That's the joy of not using create-react-app here. Everything is up in the air. And I'm probably going to break many best practices, and learn why they are best practices when something falls down. For me, that's the way I learn best. I need to try something to see how it fails before I fully understand how it works. Some people might see that as wasted effort, but I find that having a better model of the underlying systems at the core of my knowledge set always pays off later.

So I'll stick with an inremental change to webpack configruation. I'll create an "app.js" file in a new "./src" folder in the project. I don't need to put anything in it yet, but I will need a root JavaScript file. Then I need to configure webpack to build the app bundle and put it into my project. I'll create a new "dist" folder for now, which I expect to eventually usurp my "wwwroot" folder. But I'll leave the old structure intact as well, so I can still publish the site while I work. Sometimes it's easier to keep a paralell process running while you are making a change, and then do the switch-over when you're sure that both work. That can lead to extra effort and interim configuration, but it means the site stays up.

Since I'm using webpack-dev-server, I automatically get the package built when I run my test server. But I also need to be able to build on demand as well. Let's add a new script to "package.json" to build the webpack bundles:

At this point we haven't made any actual changes to the visible site, except for this description of the change. But we're better set up for starting to break down the blog entries into individual files, and for writing JavaScript for the site. The next question, however, is how much client-side JavaScript are we actually going to need? It seems like the majority of the work is going to be server side, putting together HTML pages made up of the blog entries. The client need know nothing of that. So that tells me it's time to start looking at server routing, which means it's Node.js time.

Site Templates and webpack

The next step is have an "index.html" file created in the new "dist" folder during the build. We could copy an existing file, or do lots of other manipulations. But there's a plug-in we can use to do that for use, "html-webpack-plugin". So let's add that to our project, and see what happens.

I'll just pretend not to worry that this added 155 packages to my project. I hope we can all accept implicit dependencies. In this type of coding, we really have to. It goes against many of my in-grained biases after 20+ years of professional coding. I like to know how everything works. In this case I won't even learn the names of those 155 packages, much less what they all do or how they do it. But there comes a time when you must trust the mental model in your head, and cut off a chain of complexity in order to make progress. So yes, I "only" added html-webpack-plugin"...

At this point, I want to digress a little bit. The naming conventions in all of these various components is a wonderful example of where a solid style guid would be useful. We have JavaScript, with two caps. webpack, with none. Some things are camelCased, while others-use-hyphens. It's hard to keep track of what to use when. I'm sure half of these blog entries are inconsistent. But that's what happens when we have code using mulitple plug-ins from multiple developers. We lose some consistency for the sake of allowing a large number of contributors to make our lives easier. This is always part of the battle with shipping software from a large company. You want a consistent style guide, but you don't want to hamstring developers with an onerous overisght process into naming. Because we all know there are three really hard problems in computer science. Naming things and counting.

Now let's use this plugin to generate our index.html. According to the documentation, I should be able to add it to my webpack.config.js file and have the index.html file generated. Let's try that.

This works, except the "index.html" file that was created was added to the root of the project. I need to configure the output folder.

Now I've got the index.html file building. After changing the webpack.config.js file to use the "dist" folder as the contentBase, I get a single blank page named "Webpack App". I obviously need to do some more tweaking of the configuration, possibly using a new template, so that I can get a proper title and some of the actual site content in there. The title is easily changed with a property, but for React I'm going to need a >div< for the content to render in to. We don't need anything fancy, so we'll put a default "index.html" file tempalte into "src/templates". I'll keep this basic for now, but we'll likley want to take advantage of bootstrap for some layout goodness. We'll pull the header from this page, and consider this a partial refactoring towards getting the page in "dist" to look exactly like this page.

You can see the index.html file contents by viewing the source in your browser window. I'm doing nothing fancy yet. I'm not even using a templating engine, just raw HTML. There's a nice >div< where the React content can go in. Next step is to write some of that content. The simplest thing would be to generate the "Nav" component to map the exisitng (and barely functional) navigation links on the page. I'll do that in raw JavaScript and JSX, and then look at converting everything to TypeScript once that's working. Let's prove the initial concept, and then move forward on a cleaner language definition. One problem at a time. If I tackle them both, I won't know which has my errors when it doesn't work. Because, as a programmer, you always assume something won't work. Then you can be pleasantly surprised when it does.

We're nowhere near parity yet, but pieces are coming into place. Another hour another step closer to a functional website. It would go a lot faster if I wasn't documenting this whole process, but where would be the fun in that?

To Serve and React

Let's start with the simplest React component, a single string. We already created an empty "app.js" in our previous iteration. So it's the perfect place to insert a simple component. We'll stay away from TypeScript and JSX/TSX, and just validate that our build process is working as expected.

We need to import React and ReactDOM, and then we create a root element that is a div with a heading. Finally, we render that into the existing document element div with the id of "root". Let's run that through "npm start" and validate that we can compile React.

I added the "react" module a long time ago, but never added "react-dom". A quick "npm install -D react-dom" and we're good to go. The page now says "Test" in pretty large H1 letters. So let's clean up a bit and do an interim check-in.

I like "app.js" (probably to become .tsx at some point) for the main app entry point. So I should use that for the div id, instead of "root". That requires a change to the "template/index.html" and the corresponding change in "app.js". Looking at the template is when I realize that the html-webpack-plugin is pretty much superfluous at this point, since all it's doing is adding one line to my template. A line I could easily add myself. But there's no harm being done, so I'm not going to spend the time now to back it out. I may find utility in it later. If I was using an issue tracking system for this project, I'd enter a backlog item to go back and investigate this later. Which now makes me think I should start using an issue tracking system. I'm already using Visual Studio Online for this project. Which means I have such a system in place already. I'll create the backlog item, check in at this interim point, and then come back and start looking at making a real React component using TypeScript.

Time for TypeScript

Now that we have JavaScript code compiling with React, it's time to move to TypeScript. I'm not a strong typing zealot, but TypeScript adds a lot of safety and good syntax to JavaScript that I want to take advantage of. So let's move our core application to TypeScript. To start with, we only need to worry about changing our "app.js" to "app.ts", since TypeScript is a superset of JavaScript.

I began with the obvious step of renaming the file and changing the entry point for the webpack build. And, surprisingly, everything appears to work correctly. But is that actually the case? Does the current version of webpack have built in TypeScript loader support, or am I just lucking out because the contents of the file are JavaScript? Let's rewrite our "app.ts" in a more TypeScript format and see if it still works...

...and it really doesn't. Not very well. I need to have a webpack loader for TypeScript, especially if I want to use .tsx files. So it's time to use a loader. There are a few choices, but I'm going with "ts-loader". There's "awesome-typescript-loader", but I tend not to trust software that has to tell me it's awesome. I'll stick with waht's in the TypeScript documentation. Again, most choices are fine, I'm just picking one based on any criteria that helps me move forward. "npm install -D ts-loader". Another 122 packages installed. Also, the tutorial I'm looking at recommend adding "source-map-loader". Knowing that debugger is always easier with good source maps, I'll grab that too.

Now that I'm starting to use TypeScript, I also need to configure TypeScript. That's done with a "tsconfig.json" file at the root. I also need to tell webpack to use this loader in the webpack.config.js file. I'm following this tutorial, so I'll use the contents from there.

At this point, an awful lot of debugging ensued. I'll spare you the full dump of all of the outputs, but here are the hightlights:

Now that all the debugging is done, we have a lot of new configuration but the core processes haven't changed. And that allows our "app.tsx" file to look a lot cleaner than it did before. Here's a simple React component displayed on a page:

Given the large amount of redundant code we had before to put a simple React component on the screen, this is a great improvement in syntax at not much cost in configuration complexity. We can now begin to build components for the web page for the Navigation Bar and an individual blog article. We can then convert the single HTML page we have to a better formatted blog article system using React components. Once that's done, we can start thinking about how to best display the blog entries, which will likely involve building pages on the fly on the server back-end. We'll need an API for getting sets of articles rather than the full stack at once, so that likely means we'll need to look into routing, and possibly Express. And maybe Redux for storiong our current article state on the client. So some exciting ground to cover in the next few installments.

Let's look at the application we've written so far. And I'm using the term "applicaiton" loosely. What we have done is write some TypeScript that gets compiled into a file called "bundle.js". There's a file called "index.html" that refers to bundle.js. We've set up webpack-dev-server, but we really don't need it. We can open the HTML file and, if bundle.js is next to it, we see the content in the JavaScript bundle. Everything is running on the client, so we effectively have a static site. While it's slightly more complicated than the single HTML page I'm using to document the build process of this app, it has none of the content in it. I'm eventually going to want to turn these blog entries into entities that can be served, and have them appear in an app with full search. We'll want partial download, only showing the latest entry, and a way to iterate over the entries. That's all going to require a server application, which is where express is going to come into play. I'll have to dice these entries up into a format that the server can use to make a better experience. Everything we've done so far is either client-side code or configruation of the development environment for writing that client-side code. We don't want a blog that requires downloading every entry before you can see the first one. And that's what we have right now.

If you're looking at this process and finding it too slow, this probably isn't the tutorial blog for you. There are lots of "fast" tutorials out there, but I find that they don't fully explore the concepts or the reasons why things are set up the way they are. I'm being pedantic so that it obvious exactly what each line of code does. There's a git commit for every blog entry (sometimes a couple). At some point, the commits will be linked to the blog entires, and you'll be able to see the progress in the repository.

Most importantly, I'm not suggesting that this is the ideal way to write a React application. Far from it. Use "create-react-app" and as much help as you can get. However, I find that unless I understand the principles behind those helpers, eventually I hit an issue that I don't understand. Frameworks are wonderful, but they hide details and force design decisions on you that often you aren't even aware exist. In those times, having a grounding in the basics is invaluable in debugging. So I hope this helps you understand the underlying technology as much as it is helping me cement what I know. In the meantime, I hope the huge growing page of documentation of the process of building a React app from scratch is at least some use until we get to the point where we document making it a lot better.

Our first React Component

I have to admit at this point that I'm getting tired of writing these entries into this one huge HTML file. I'm really looking forward to the point wehre I can start a new entry in a folder somewhere and the system will automatically build the necessary infrastructure for displaying the content. Then I won't have to worry about accidentally modifying the content or the effects of full-file reformatting. I also feel like the page is incredibly unwieldy and hard to read. So I hope that the transition away from this file make things better for any readers. Of course, with no commenting features and the fact that I don't have an email address hooked up on this page, combined with a complete lack of analytics, I have no idea if anybody has even looked at this page besides me. Maybe we'll look at analytics soon. But I'm going to bang ahead and "do the next right thing", which is my persnoal distillation of Agile methods in this kind of informal context.

Let's componentize the NavBar that was added several iterations ago when we added BootStrap. At the top of the HTML monolith, we have a block which displays a simple, and current non-functional, navigation bar. We didn't need this, and still don't really, because there is nowhere to navigate to. But I added it so I'd have a place to play with BootStrap formatting. And it makes the simplest piece of HTML we have to componentize, so we'll start there. Let's add a new folder, "components", and add a file for "NavBar.tsx"

This looks a little different from the Test component we had in app.tsx, so let's unpack this. The major change here is the use of a React Stateless Functional Component rather than a class. In the case of our NavBar, we have no state in the component. The only reason to use a class is to store state. So instead we can use a simple function for our component. We use the "React.SFC" type as a base for our stateless function, which allows the compiler to infer the correct types on our properties. Which we're also not using any of yet, but we'd like to use the standard pattern and definig the type as an "SFC" declares our intention. At this point JSX/TSX syntax takes over, and we can simply copy the NavBar HTML into the return statement. Now that this component is in its own file, we also need to make sure that it can be seen by the other code. So we "export" it.

For the most part, this is fairly elegant syntax. There are a few warts to point out. First of all, I'm not impressed by the need for the '(' after the return statement. JavaScript has a feature known as "automatic semicolon insertion", where the language tries to be "smart" about the code it reads. There are several cases where the compiler infers that you means to terminate a statement even when you explicitly didn't. A "return" statement on a single line is one of them. So we have to use the '(' to tell the compiler to keep parsing the return statement on to the next line. This seems like a poor choice in design, but it's so ingrained in existing scripts that it will never be changed. So we all need to be aware of it.

Another wart is the use of "className" as the HTML attribute used for setting the "class" of HTML elements. The designers of React decided that they didn't want to have the ambiguous name "class" in the JSX syntax, and so they mapped it to "className". However, the HTML property underlying this is still named "class". So if you're copying raw HTML into a React component, you need to change the "class=" properties to "className=". Which again is a minor pain. At least the compiler and Visual Studio Code are smart enough to let you know there's an issue, and a quick Google search will steer you right.

Now we need to update "app.tsx" to use this NavBar component. Given that we're going to want to have the App have a number of top-level HTML tags, and React requires a single top-level tag, we need to wrap the content in a <div>. That leads me to to want to refactor yet again, and move the App into a component so that I can put that extra complexity there. So let's create a new "components/app.tsx" and rename the old "app.tsx" to "index.tsx". That means the template stays simple and the top-level entry point is clean, while the wrapping of things into multiple divs is done in the app. And we'll have to adjust the webpack config to use the new entry point. Now our entry points are all "index.*" names, which seems cleaner as well.

We now have a very clean split between components and the infrastructure of the page, which we can expand upon to focus on the display hierarchy. Now we need to start getting the actual content on the page. Instead of designing something up front, we'll continue with the iterative approach and see what develops. We all "know" we want a data store of articles that can be served up by an application. But let's forget that and just start cleaning up the content in the next iteration.

Expressionism

We're going to need to handle a variety of different kind of URL requests for different pages. I'm envisioning a root blog page with the first chunk of a number of articles, and a way to refer to a specific article. That implies that we're going to need to model a number of different views in our application. And we're going to download articles one-at-a-time, rather than pulling in everything and generating the page in client code. While we've got nothing but a few hundred lines of text so far, the vision is to eventually have hundreds of posts covering a variety of topics.

There are two models of routing in React. "React Router" is the most obviously named component, but that's entirely used for client-side routing. In the simplest terms, it's a way of having a single page application appear to be multiple pages by using <Link> elements to model HTML links on a page. It looks like your application has multiple entry points, but everything is statically loaded in a single page and React takes care of the details. I'm going to break with the Agile idea of just doing the simplest thing that gets the job done. React Router would solve my problem, but it doesn't feel like a scalable solution. We'll likley use it at some point to do some view manipulation. What I'm going to need is an API that the application can call to provide the content. This is going to involve writing a new server component to handle our API. I'm going to need some way to identify articles and query for them from the server.

Let's keep it simple, and for now state that the articles are stored with a name that is an integer that indicates the order in which the articles were written. So the first article will be "1.xml", followed by "2.xml", etc. We can start tagging the contents later, but for now we'll just make sure the articles are well-formed HTML content that we can serve when requested. Our simplest goal is to reproduce the exisitng monolithic page, so we'll define one API method called "GetArticles" that returns an HTML chunk created by concatenating all of the files together in order. Not pretty, but it's a simple starting point.

Right now everything under "src" is being compiled into our React app, so let's create a new top-level folder called "server" for the server code. We'll add a "server.ts" file that creates a node server. And we'll avail ourselves of the de-facto standard for server routing and API handling for Node.js applications, Express. Note that we'll now have both a web server (webpack-dev-server) and our API server, which will add a new set of problems to deal with soon.

In order to build the server component separately from the client code, we need to have a webpack configuration for the server. This is done by having webpack.config.js export an array of configurations. When webpack goes to build, it will build each of the configurations. We'll take the existing configuration and duplicate it, then change the module source and entry point to build the server scripts. We'll put the output into the "/dist" folder as "server.js". We should then be able to run the server using Node.js. To keep things extremely simple, to start with I'll copy a standard "Hello World" Express app. Which of course means we'll need to "npm install express @types/express".

Back to the internet for some help. The problem here is that while we need to bundle all of our resources for the web client, the server is running on Node.js. And we don't want to bundle Express into our scripts. Instead, we let npm manage packages on the Node server. It is called the "Node Package Manager" after all. So we need to tell Webpack to not pick up these dependencies. Yet another problem that has already been solved. We can use the "webpack-node-externals" package to teach webpack about dependency exclusions. "npm install -D webpack-node-externals", and define some "externals" in our webpack.config.js setting for the server. "npm run build" to build, and then we can run the server using "node dist/server.js". And we have our "Hello World" server from http://localhost:3500. Let's look at the webpack.cnofig.js now:

There is enough similarity in the two branches that I might want to consider refactoring the common settings. But I'm not going to combine refactoring with a functionality check-in. This seems like a good stopping point. We now have a client and a very basic server. Which now, as is typical, sets us up for a new raft of work. We'll need some way to run the Node.js server in a production environment, so we'll need some more configuration. We'll also need to build an actual server that returns article data. And we'll need to test that API, so we'll need some testing infrastructure.

State of the Craft #1

I'm a few posts in, so it's time to reflect on the process and what to do going forward. I have no means of commenting on the site, and I haven't even posted my name on it as this point. If you're enjoying this site, you're reading the musings of a random guy building a web site by avoiding all of the really cool infrastructure that makes building a web site early. I suppose you must be wondering what kind of a lunatic would do that. I'll add an "About" page when I get some routing infrastructure, and post some information about myself. Right now I'm enjoying the anonymity of the whole process. Once I do get some info up, I doubt I'll add a comment section, but I may add an email forwarder so you can send me feedback.

I'm also planning to get some analytics set up so I cna see if anybody is actually reading the site. This is primarily an exercise for myself in improving my skills. I've been a thick-client developer for many years, and have worked the past few years on some very custom back-end service infrastructure. But I don't get to deal with "simple" web applications at work. So I'm happy writing this for myself, exercising the writing and programming muscles. If there are people reading this, I'll likely spend more time editing the content. I'm writing everything in Visual Studio Code, which has become my default editor lately. But VSCode doesn't have spell checking, grammar checking, or much of any English linting capability. I know I probably have a few dozen (at least) typos on this page. But given that my current focus is in getting the site up and not in the presentation, I'm (mostly) fine with that.

Even though I am publishing the content with each push to the server, I'm trying to document the natural creation of a web site. And having a site is more important than spelling. I'll make a pass and go back and fix the articles later. I'll also improve tagging for the code snippets, instead of hard-coding <pre><code> sections in the document. I'd actually like to create links to the actual file revisions in the git repository, so readers can follow the progression. I've learned that Visual Studio Online doesn't support public repositories, however, so I'm going to have to move the code to GitHub or some other public site to do that. That's well down the product backlog, so for now I'll keep adding the code snippets. I'll have to start cutting them down, I don't want many more the size of the webpack.config.js in the last post. So I'll focus on code deltas.

In terms of time, I think that I'm spending slightly more time documenting the process than I am actually coding. If I was just moving forward each iteration, I'd be checking in a lot more and making a lot more progress. That's a choice that meets with my goals, so I'm okay with that. But it's something I'll continue to be conscious of going forward.

I'm also not spending any time on the audio synthesis programming I want to get into. I plan on using a similar concept there, focusing on explaining the basics in a way that I haven't seen many other guides do. I'm particulalry interested in the differnt types of signals used in analog synths, and how they interact and can be morphed into each other. Is an audio signal sound, or is it a trigger for another element? It can be both. Pretty much any in-range volatge signal can do something to the system, and the feedback possibilities in the signal chain are endless. However, to write those articles, I feel like I need to be able to embed pictures and post links between the articles. So that series is blocked by progress on the web site.

In summary, I'm having fun, and I'll keep going for a while. I hope that, if you found this site, you're finding some value in this basic approach. I'm already looking forward to starting new projects with a lot more tooling, now that I feel that I have a much better understanding of how Webpack, npm, TypeScript, React and Express work. I could probably better my progress on this site in a few minutes with "create-react-app". But where's the fun it that?

Analytics

This is a small iteration to add Analytics. For lack of wanting to research options, I decided to stick with Google Analytics to start with. This is due to it's ubiquity and it's ease of use.

The typical case is very easy. Head to "https://analytics.google.com", sign up, and sign up for an analytics account. Then you create a "Property" for your site. Google Analytics then serves up some HTML to put at the top of your pages, and you're ready to go. In this case, I added this right under the <head> element in my template file.

In my case, I wasn't a typical customer. I had an extra step to perform because my Google account is linked to a domain managed by Google for emails and other services. And analytics isn't on by default. So I had to go to http://admin.google.com and enable Google Analytics for the domain. This should have been easy, but the default view for services on their page only show 10 services, and it isn't particularly obvious that what you're looking at is a paged list. So I spent about 20 minutes trying to find the missing service to enable. Eventually I twigged on the UI, but there's a huge white-space region under the service list that is typically an indicator that you are seeing the last page of a list of data. There are a number of common UI paradigms, and Google violated one of my expectations here, costing me a chunk of time. Usually their UI is a bit more obvious. I wonder if there is less scrutiny of the design since this is an admin-only page?

Regardless, I've got some analytics now. I've left my local IP in the analytics. You can filter a range to ensure that testing and local users don't skew the data. In my case, I want to see at least a couple of local hits before I turn off the local testing. That way I'll know the analytics are working. When you don't have testing, the next best thing is feedback. So I'll watch the analytics for hits from me here, and then I'll filter myself out, hit the site a lot, and then make sure it doesn't pick them up. I remember when software companies used to actually hire people to test software. I've worked with some great testers over the year, and I think that the switch to have developers test their own code has been a partial mistake. We need to write good tests, but having people paid to be in the mindset of breaking things allows for a whole differnt raft of things to be discovered in the software. We all make assumptions, but a true tester validates every one of them, every time they see a new build. QA really should be a cornerstone of development, and is a strong partner to an Agile process.

Swapping out the server

Until this point I've been using webpack-dev-server as my development server. There are some nice benefits for doing so. I get automatic hot-loading of React component updates as I save files, and I didn't have to start with a server of my own. But now I've got a server of my own, and continuing to use wepack-dev-server means I need to find a way for that Node.js server to coexist with webpack-dev-server.

There are a number of tutorials that purport to provide a great experience of having webpack-dev-server work alongside your Node.js server. However, after reading a number of them, I have come to the conclusion that I really don't think the extra configuration headache is worth it. To use both, I need to have webpack-dev-server proxy to the Node.js server to avoid cross-site-scripting security measures in the browser prevent the app from working. I have to have separate configurations for produciton and development that are wildly different. And I need to run two server concurrently and deal with all the complexity.

After considering the options, I decided instead to forgo webpack-dev-server, at least for now. First, the complexity of the proxying and the large delta in configurations between produciton and development doesn't feel good to me. It feels like a "code smell" that I want to avoid. Secondly, I really don't feel like I need hot reloading. I've spent a lot of time working in the compile-test loop in native languages, and having the code change under my running instance of the app feels wrong. I may like hot reloading, but I don't need it. And now that I have a server, why not use it for serving up the static React content as well as the API results?

This leads me to the following flow. I have client code (using React) that builds a static content bundle that represents my Single Page Application. This gets built into the "dist/public" folder. I then have a Node.js server that gets compiled into a "server.js" file. To run the application, I need to run "node server.js", which creates a server. This has a default route that returns the contents of "index.html", and refers to the bundle.js in "public". So the application works at the root URL, and is in a great position to add routes for the API.

This just feels a lot cleaner to me than trying to use webpack-dev-server and Node.js. The amount of research for this step took a couple of hours, but the code changes are pretty small. I change the "start" script as follows

In "webpack.config.js", I changed the build location of the client code to "dist/public". I also deleted the "devserver" property, since the client doesn't use webpack-dev-server. And to reduce our dependencies, I also did an "npm uninstall webpack-dev-server". This was a dev dependency, so it didn't build into my outputs, but this removed 140 npm modules from my install, so it feels like a useful cleanup step. The only tricky part in "wepback.config.js" was in the following property I had to add to the server build:

This is a counter-intuitive setting for me. "false", in this case, tells Node to use the default behavior, which is to set __dirname to the folder of the "output" file as specified in the webpack config. "true" uses the path of the input files, which I'll never want when the input files are TypeScript and have to be compiled. So this took a few minutes of debugging to understand. But it's a clear and precise change, and seems liek an artifact of a design decision in webpack rather than a hack.

The final change is to update the "server.ts" code to handle more routing. I need to add routing for the new static files in "dist/public". The default route needs to return the contents of "index.html" so that the app loads. And since I'm now down to a single server, I'll use a better default port that has a chance of surviving through the deployment process. Here's the update:

So not a huge amount of code change, but a nice platform for starting the API work next time.

Code means Tests

A quick aside on analytics: I had one viewer from Houston. Hello. Hope you found it useful. This is about why I expected, since I have no SEO and a single page with no links. Which frees me up from worrying about the details for now. Let's continue the work to make something readable, without the pressure of a lot of readers.

It's time to add an API to get article information. And that means code running on the server that isn't complete boilerplate. And, to me, that means I need tests for that code. So it's time to pick a server-side test infrastructure. Ideally, it would be something that allows me to test the server and the client with similar syntax, and allows me to write the tests in TypeScript. Technically I can use any framework with TypeScript, since it compile to JavaScript, but built-in support may be clearer. Plus I'd like to have up-to-date type files so that I get syntax coloring in Visual Studio Code.

The obvious choices are Mocha and Jest. Mocha is the established favorite, and Jest is the now-mature-but-previously-new-hotness. One thing that Jest has going for it is direct TypeScript integration with the "ts-jest" package. It's also what many teams are using at my full-time workplace, so learning Jest makes the most sense for me. That's enough to make the choice for Jest for this project.

"npm install -D jest ts-jest @types/jest"

From the NPM page for ts-jest, we grab the following to add it to our package.json:

      package.json
      "scripts": {
    ...
+       "test": "jest"
      },

      "jest": {
        "transform": {
          "^.+\\.tsx?$": "ts-jest"
        },
        "testRegex": "(/__tests__/.*|(\\.|/)(test|spec))\\.(jsx?|tsx?)$",
        "moduleFileExtensions": [
          "ts",
          "tsx",
          "js",
          "jsx",
          "json",
          "node"
        ]
      }

We have now wired up Jest and can run npm test to run our tests. Doing so, we see that no files match our definition of a test file, so we match 0 tests. Tests fail. Which is a good thing. The ideal way to write "Test First" code is to write a test that fails, and then write code so the new test passes and all of the exisitng tests continue to pass. I'm a huge fan of Test-First programming. I also work on a lot of legacy code, and much of that is written in a way that makes it very hard to isolate dependencies and test the code. So I don't use Test-First methodology as often as I'd like. Since I'm controlling the whole stack here, I'm building tests in fairly early.

Some might say I should have started with tests, but I'm not a complete purist. I don't like writing a lot of tsts for boilerplate code that might change, or code that simply invokes underlying infrastructure. If I'm using a library, either I need to trust it enough that I believe it does what it says or I shouldn't use it. So I assume those libraries are tested, and I don't need to ensure that a call to a library does what it says it does. The whole point of a Unit Tests is to test my code, not the libraries I call. I need to vet what I call, choosing known frameworks and trusted repository providers, but recent stories of hijacked libraries and malware injected into npm packages show that there is always a risk. The question is, how much risk do you take versus the burden of writing every piece of infrastructure yourself? For a large corporation, there should be corporate validation procedures and oversight. For this kind of solo project, I'm adopting a "trust but verify" mentaility. I'm being careful what I include, but if somebody hijacks this site right now I have no user data, no cookies, and no database. If you can hack this page, you've hacked this page. Congratulations. But it's not worth worrying about at this point.

After that digression, let's pick up the red-green test cycle. We have no tests, which is a failure, so we're "red". I want to have a passing test run. That means I need a test. One that succeeds unconditionally. I'll delete it as soon as I have "real" tests, but right now I want that "green". So let's generate a server-side test that passes.


    server/__tests__/server.test.ts
    test('true', () => {
      expect(true);
    });

And we have a working tests. 1 found, 1 passed. So we can start writing the actual API.

Rethinking the Server

The previous plan was to build a web API that would make it possible to query all of the articles and do a request-time generation of the meta-data needed to drive the client software. However, I've been away from this project for a couple of weeks due to other commitments. And while I haven't been writing any new code on this project, I've been thinking about the server. And I'm quickly beginning to believe that I don't need one.

What I need is a way to get meta-data down to the client. My current publishing plan for this site, once I get off of the single page, is to have the articles be stand-alone Markdown encoded documents that I can write in a more full featured editor. Once I've added a new article, I'll Webpack up the site and FTP it to the final location. So the content is actually constant after each publication. Which begs the question of why would I generate data at run-time that I could more easily compile during my Webpack process?

If I parse the articles and create an index database file in JSON, I can then have the client make a simple HTTP GET request for the file and populate all of the metadata with a simple call. Drop this infoormation into a store like Redux that is designed to work with React, and we can populate the blog off of that. Plus, there is no need to write tests for both the client and the server. And it's easy to create some JSON data for tests that can drive the system, and simply change the source of the JSON at run-time to make the site active. It means I'll need some scripting to build the JSON store, but that should be easier than writing an entire server.

So the next step will be creating a simple meta-data format and getting that to display a simple test page. Which means I can build up two content hierarchies, one for testing and one for the site itself. I'm going to leave the client-server split in place, however, since I want to prove I don't need it before I remove it. I could roll back the project, but I'd rather keep this a linear record of the design to better reflect the realities of working on a project that evolves. So time to create some client-side unit tests, some test content, and some actual code to make the tests pass. That'll be the next iteration.

Back to Webpack Dev Server

The first thing I realized as I started writing code for prototyping the article list code is: 'I want WebPack Dev Server again". Every time I make a change, I have to CTRL-C the running server, "npm start" and then reload the browser. Whereas earlier I had everything set up to automatically re-build and refresh the browser after every file save in Visual Studio Code. I said a few articles ago that I'd try without this and that I probably didn't need it. I was wrong. Need is too strong a word, but I'm already tired of the extra steps in my inner dev loop. So before I start talking about the code, I need to restore the old setup.

We don't have a server, so I can remove the entire "server" folder and the tests wihtin it. I can also remove the entire server build section of the "webpack.config.js" and restore it to it's old version. We'll do that with "git checkout 21a35a3490c5c5f03ee89af12a0aec8981fe55d3 webpack.config.js". I used the VS Code Git Extensions to get some nice utilities to view git history, found the revision for the file I wanted, and then git checkout can restore that version. We can also diff the "packages.json" file to find out what needs to go. It appears I need to reset the "start" script to use webpack-dev-server, and remove Express and webpack-node-externals. "npm install" to reset the dependencies. "npm start" and test it out. We've got webpack-dev-server with auto-reload on code changes working again, and a lot less code. And no tests. But it's a lot easier to prototype.

Since I'm doing this project in 1-2 hour chunks in my spare time, scattered around a bunch of other projects, it's very easy to find myself changing direction and learning things in the intervening time that make me question earlier choices. While I haven't written a lot of code, I have spent a lot of time on infrastructure and documentation. I'm gratified that, having made each change methodically, it's easier for me to understand the changes I need to make to follow my prime tenet of coding, and life in general: "Do the Next Right Thing." It doesn't have to be the final thing, just the next thing. The step that moves you closer to either completing a goal or understanding what you need to know in order to mvoe forward. It's very easy to get caught up in perfectionism. But by willing to make a choice, move forward, and learn from that choice, you can make future steps easier. In this case, ended up where I started, but with a better appreciation of webpack-dev-server. And a lot better understanding of React app state. Which I'll get into in the next article. I'm going to commit the changes to get me back to webpack-dev-server now, and then I'll talk about the code changes I made in React that led me to this decision in the next article.

CraftyProgrammer.com