Eliot Jones

Show me the information

Eliot (Eliot Jones) — Sat, 21 Mar 2026 00:00:00 GMT

Since I'm in an unhinged rant frame of mind I wanted to also rant about this tendency for 'minimalist' thinking in UX/UI. I'm going to blame Google and their Material UI for this, they should be blamed for most things, but the trend has become something of an accepted wisdom amongst UI designers.

The idea is you keep the UI nice and 'clean' and only show the user the fields you believe are relevant to them. For example you truncate a message because it would wrap and break the UI scaling. Or you show just the date part of a timestamp, or even worse show "a few hours ago" instead of the actual date. Maybe you show the full time on a mouse-over event, but how do I copy that?

Hiding information to make your site visually appealing breaks real-world tasks. You can't predict what the user is going to do, you are not an oracle, so just show them the information. Even if they decide to put the entire text of the Iliad in a description field. It'll look garbage but at least they can complete their task.

There is a lack of empathy for the user in these choices. What is your user trying to achieve? How can you make that task as easy as possible? What about users who fall outside the flows you predicted they would need?

First up there's this screenshot from the fundamentally cursed Gmail UI:

I've redacted my personal information from the screenshot but the To field only tells you the first name of the recipient. What if you have multiple recipients with that name? Why not show me the email address next to the name? I needed to double check the address because I was sending an email from a different device outside of the Gmail ecosystem. You can get to the email address by clicking the expanding chevron next to the name but this is another speedbump on a common task. You tell me my own email address just above, why not the recipient's? Well it would scale badly for multiple recipients, but that's your job as a designer. Don't just hide the damned information. Systems should be designed to do things.

Then the worst offender. My bank. I was sending money abroad for some purpose using a wire transfer. The bank listed 2 options for the recipient bank that differed only in the last 3 letters of the SWIFT code. The recipient name, address, account number, etc., was identical. Because the first small test transfer I sent opted for the wrong SWIFT code leading my money to disappear I had to try again with the second SWIFT code. The second transfer worked so then I needed to send the remaining payment amount. But because of the idiotic decision to hide the full payee details I couldn't actually tell which of the 2 existing payees was the right one, so I then had to add the details a 3rd time. That left me with this problem:

Again I've redacted information here, these aren't real account numbers or names but I kept the redaction consistent. As you can see there's 3 payees with completely identical listings. If I send money to one of these my money disappears forever, lost somewhere in the international wire system. Well, you think, they've probably just done an inconvenient information hiding thing again. Maybe just click click the 3 dots next to the payee and you'll see the full details?

Nope, the only options there are to delete the payee or send a payment. If you try sending a payment there's no further information about who you are actually paying.

Granted this is a bank UI so their systems are probably ancient and they don't have the time or software development resources to go in and add a way to retrieve the information like Google has. But this happens because the designer and their poor taste wins out over the needs of people who actually use the systems they build. I'm sorry that the information density or length or shape is inconvenient to your artistic vision. You're wasting my time and potentially losing me money. Just add the damned fields to the UI. You have them, they're in your database somewhere. So show them to me!

Modern UI design is cursed. People with good aesthetic taste but terrible taste in actual usage win out over the developers and users of systems all the time. There is a lack of empathy. I feel like the only reasonable person in a world gone mad.

Cheese sandwich

Eliot (Eliot Jones) — Fri, 20 Mar 2026 00:00:00 GMT

If you are the type of person with Very Strong Opinions about the (negative) health and moral implications of plant-based 'meat' you are signalling that you are fundamentally intellectually incurious. You have an opinion you haven't taken the time to inspect and have adopted certainty without rigor.

This is a picture of a cheese sandwich, I made it at home, it was ok:

And this is the ingredients list of that cheese sandwich:

Lettuce
- it's a lettuce
Tomato
- it's a tomato
Balsamic vinegar (a small drizzle rescues supermarket tomatoes from their cardboard nature)
- wine vinegar
- concentrated grape must
- cooked grape must
- coloring E150(d)
Mayonnaise
- rapeseed oil
- egg
- spirit vinegar
- pasteurised egg yolk
- lemon juice from concentrate
- sugar
- salt
- mustard extract
- paprika extract
Cheddar cheese
- pasteurised full fat milk
- vegetarian rennet
- salt
- starter culture
Bread
- wheat
- rye
- barley
- oats
- millet
- corn
- sesame seeds
- sunflower seeds
- linseed
- soy
- 100% vegetable shortening
- sugar
- molasses
- salt
- yeast
- calcium propionate
Pickle
- carrot
- rutabaga
- onion
- cauliflower
- sugar
- barley malt vinegar
- spirit vinegar
- salt
- tomato puree
- dates
- rice flour
- apple pulp
- modified maize starch
- sulphite ammonia caramel (color)
- onion powder
- lemon juice
- spices
- colouring food (barley malt extract)
- herb and spice extracts
Add ham to taste.

Now, is this sandwich the healthiest food in the world? No, not really. But I hope you'd agree it's not complete poison either. Most of the ingredients list are recognizable things. Except rutabaga, I have no idea what rutabaga is, but Wikipedia tells me it is turnip. Importantly, many of them are things that have been through multiple processing steps. That's just what we do with food, we make ingredients into other things. We've been doing this for millennia.

What is spirit vinegar? Per "The Vinegar Institute" (which sounds fun) all vinegars start out as agricultural sugars that are first fermented into alcohols, followed by the alcohol being fermented into acetic acid by other bacteria. What about modified maize starch? Modified how? Sounds pretty suspect. Well, the website "Starch in Food" (I'd bet a 'The Vinegar Institute' and 'Starch in Food' get-together is a real party) says it is a starch extracted from some kind of grain or vegetable treated to improve its uses in food in one of three possible ways:

Cooking/roasting
Enzymes
Chemicals

The packaging doesn't specify which of these 3 methods was used and it doesn't really matter. I'm fine eating this stuff. A cheese sandwich is full of chemicals. I'm full of chemicals, you are full of chemicals. Toilet bleach is chemicals, so is a lettuce. Food containing chemicals tells you nothing more than whether or not the food is a vacuum. Which brings me onto the topic of my rant.

If you have strong opinions that plant-based meat substitutes are unnatural or unhealthy solely on the basis of their ingredients list you are being incurious and I'm sick of hearing about it. These days you can't move for people who have Opinions that they can't wait to tell you if you order one of these things. And I don't think these people have come to their Capital-O Opinions organically; and I don't think they've inspected why they think what they think. And that's intellectually lazy.

In general there are different reasons someone might choose to adopt a vegan diet, vegetarian diet or simply choose to eat less meat. Each person will have their own reasons.

Health reasons.
Moral reasons, a piggy is much like a cute little doggy but we only raise one of the two in horrendous conditions before murdering it and chopping it into pieces. (The dairy and egg industry are, if anything, crueler, but in life you generally have to decide at what point you're making the perfect the enemy of the good, and that decision is up to you)
Environmental reasons, the environmental footprint of meat (and dairy) is generally larger than vegetarian foods.
Taste, some people don't like the taste of meat.

Strong Opinions on plant-based meats are usually one of either:

These taste bad and aren't really like meat. Which, sure, y'know, I think olives taste bad, but I'm not annoyed by the olive section of the supermarket. Just don't eat them.
These are frankenfoods. They are fundamentally unnatural. They are extremely unhealthy, made in a lab, and you, as someone who eats them, have sinned. These foods will kill you and my real meat burger or sausage makes me not only a moral paragon, but someone who will outlive you because I have the true science.
There is a 3rd, weaker, claim, often from other vegans or vegetarians. This is that you shouldn't need them and as a vegetarian you should be happy with only chickpeas, lentils, black beans or tofu for every single meal for the rest of your life. This usually ties into the health concerns around them. If you don't like them and you're happy with pulses forever that's great for you. But try to be a bit less prescriptivist about this please.

The health claim is the main one, and the one that has done the greatest damage to both the industry and the consumer demand for these products. We've already seen how a fairly mundane sandwich starts to sound like it was produced in an RV in New Mexico if you start to deep dive into how we process and transform ingredients.

But my response would be two-fold. If we concede the point, let's assume these ingredients are unhealthy, well... so what? It's irrelevant. I'm not eating a burger, hot-dogs, chicken nuggets or sausages to be healthy, whether meat-based or plant-based. I'm having them slathered in sauces and cheese, with fries and onion rings. Because that's nice food, not for every day, but it's a cultural mainstay of my food culture. Some days you can have pasta and sauce, some days, too many days, you can have various lentil stews or curries, but some days you just want a damned burger. Some days you just want some junk food. But these options give you junk food without the bit where an animal spends its short life in abject misery before being slaughtered in terror.

But I'm not conceding the point because I don't think you have standing to make it. I have very little confidence in strong claims made about dietary science by lay-people.

It used to be that fat was the enemy, and then it was hydrogenated versus non-hydrogenated fats. Eggs were going to kill you but now they're maybe ok. But actually probably sugar will kill you. Now maybe seed oils will, or won't, or will only do it if you heat them beyond their smoke point. Maybe sweeteners protect you from the ill-effects of sugar or maybe they kill you in an even worse way. Are nitrites in your bacon going to kill you, is red meat? Are plant-based meats? Does a glass of red wine help my heart? Does dark chocolate? Is frying everything in beef tallow the answer? How are you going to live forever?

Dietary science seems to me to be the weakest field of science, how do you pull out effects from the noise of everything people eat and interact with in their entire life? How do you get good enough source data from self-reported food diaries? How do you run big enough studies? This isn't an anti-science rant, we just need to do better science.

But pop-science beliefs from dietary science always follow the same pattern. Some pre-print or university press release, or crank self-published article, gets picked up by the media and through word-of-mouth, or funded amplification by lobbyists, becomes the villain of the moment. It's easy to understand why this happens, sickness is scary, mortality is scary. The idea that there's some key to avoid it, some guidance that can become a moral framework is extremely alluring. Just take a shot of apple cider vinegar each day, just avoid sugar, just avoid sucralose, just avoid anything with chemicals, avoid margarine, avoid butter, avoid anything with more than 20 ingredients. These beliefs provide frameworks to manage the terror, a way to assume superiority and believe that you won't be struck down, that that only happens to those who err and don't follow the true path.

Eat what you want to eat, I'll eat what I want to eat. Dietary science points us in a general direction. You probably want as much fresh fruit and veg as you can reasonably stomach or afford. You probably don't want too many carbs and only sugary things on occasion. But you also exist under a food system that prioritizes high satiety sugar-packed junk, so some days you're going to eat an entire sharing pack of sweets lying on the sofa, as the darkness gradually falls and you can't be bothered to get up to draw the curtains or switch on the light, because this is all so tiring sometimes.

All I ask is that you're not running around spouting off when you don't have a solid foundation for doing so. We can all go and find our own pet study that tells us what we want to believe. We can use that for reassurance.

But we don't have certainty on any of this stuff yet. So let's stop pretending otherwise.

So you want to parse a PDF?

Eliot (Eliot Jones) — Sun, 03 Aug 2025 00:00:00 GMT

Suppose you have an appetite for tilting at windmills. Let's say you love pain. Well then why not write a PDF parser today?

The ideal world: how the specification should work

Conceptually parsing a PDF is fairly simple:

First, locate the version header comment at the start of the file
Next you need to locate the pointer to the cross-reference
Then you can find all object offsets
Finally you locate and build the trailer dictionary which points to the catalog dicitionary

Introduction to PDF objects

A PDF object wraps some valid PDF content, numbers, strings, dictionaries, etc., in an object and generation number. The content is surrounded by the obj/endobj markers, for example a simple number may have its own PDF object:

16 0 obj
620
endobj

This declares that object 16 with generation 0 contains the number 620.

A PDF file is effectively a graph of objects that may reference each other. Objects reference other objects by use of indirect references. These have the format "16 0 R" which indicates that the content should be found in object 16 (generation number 0). In this case that would point to the object 16 containing the number 620. It is up to producer applications to split file content into objects as they wish, though the specification requires that certain object types be indirect.

Finding the cross-reference offset

To avoid the need to scan the entire file, PDFs declare a cross-reference table (xref). This is an index pointing to where each object in the file lives.

Each file ends with a pointer to the cross-reference file:

<< %trailer >>
startxref
116
%%EOF

This tells the parser to jump to byte offset 116 to find the xref table (or stream). In theory this pointer is right at the end of the file, according to the specification:

Applications should read a PDF file from its end. The last line of the file contains only the end-of-file marker, %%EOF

Though the specification says the %%EOF marker should be on the last line, in practice, things are much messier. For example, Adobe Acrobat only requires it to be within the last 1024 bytes. In real files it can appear anywhere.

In addition files encountered in the wild lacked a linebreak before the offset declaration, or had a typo, e.g. startref.

Let's assume you're able to find the declared cross-reference offset for now.

Finding all object offsets

At the specified offset you should find a well-formatted xref table:

xref
7 4
0000000000 65535 f 
0000109882 00000 n 
0000109933 00000 n 
0000140066 00000 n

After the xref indicator appears, followed by a line break, the first object number and count of objects in the subsection are given. This means: start at object 7 and list 4 objects. Each line gives the byte offset, generation number, and status (n for in-use, f for free). From this, we know where to find objects 8-10 in the file.

So in the example above -- skipping the free entry for object 7 -- this xref table tells where to find the following objects:

Object 8 (generation 0) at offset 109882
Object 9 (generation 0) at offset 109933
Object 10 (generation 0) at offset 140066

Note: files can have multiple xref tables or streams, linked by /Prev entries in their trailers.

Locating the trailer dictionary

Finally, above the startxref marker, you'll find the trailer dictionary. This provides key metadata, most importantly, where to find the root object. Once you have that, you can follow references and begin interpreting the content.

The real world: where your pain begins

Assuming everything is well behaved and you have a reasonable parser for PDF objects this is fairly simple. But you cannot assume everything is well behaved. That would be very foolish, foolish indeed. You're in PDF hell now. PDF isn't a specification, it's a social construct, it's a vibe. The more you struggle the deeper you sink. You live in the bog now, with the rest of us, far from the sight of God. If your parser expects files to obey the specification it will fail and people will think it is broken and pitiable. They will think that you are very silly.

The challenges of locating the xref pointer

We've already mentioned a few unexpected ways locating the pointer to the first cross-reference can fail:

It is not at the end of the file, nor within the last 1024 bytes of the file.
It is misspelled.
It is not in the format you'd expect.

But assuming you find a pointer, that's where the real fun begins. Because the pointer is not your friend, it is the first lie, you don't appreciate how deep the rabbit hole goes. In screening 3977 files taken from the common crawl corpus at random we detected 23 files with a bad xref declaration. This works out to a roughly 0.5% failure rate in the sample set.

PDF content starting at a non-zero offset

In these files the startxref pointer is incorrect due to a non-zero PDF content start.

This happens when there's junk data before the %PDF- version header. This shifts every single byte offset in the file. For example, the declared startxref pointer might be 960, but the actual location is at 970 because of the 10 bytes of junk data at the beginning:

ten bytes!%PDF-1.4
%âãÏÓ
4 0 obj
(content follows)
endobj
% more content
xref
0 5
% ...
<< >>
startxref
960
%%EOF

In order to adjust for this you should capture the offset of the version header in your file. If the first pointer is incorrect you should also try the offset of the first pointer plus the content start offset. But you still need to check both.

This problem accounted for roughly 50% of errors in the sample set.

The pointer is in the middle of the xref table

For some files there is no content preceding the version header, however the pointer is still wrong and it points inside the xref table content at a random offset.

For example jumping directly to the specified offset takes you to this position:

endobj xref
0 246
0000000000 65535 f 
0000184481 00000 n 
00000<---

This was the case for roughly 5 files in the error set.

The pointer is 'close' to the xref

Similar to the previous error, here there is no version header offset but following the pointer takes you 'almost' to the xref. The most common cases were to be off by a single whitespace/newline, or in the endobj marker of the previous object:

endobj
--> xref
0 4

or:

-->endobj
xref
0 7

The pointer is correct but the xref offsets are incorrect

Sometimes the pointer correctly jumps to the xref marker but if you parse the object offsets from the table they are incorrect. The table offsets can also be incorrect when the xref offset is also incorrect.

It can also be the case that offsets are correct for some objects in the table but wrong for others. This was the case for file 0002544.pdf in the sample set which had an initial pointer off-by-7. The locations in the xref table's first subsection were correct, then offsets that were off-by-4 bytes for subsequent subsections.

The first pointer is correct but the previous offset is incorrect

When a file has been modified the file's trailer (or xref stream dictionary) can contain a /Prev pointer. This is used to construct a chain of xref tables and streams. Several files had correct initial pointers however when parsing the trailer's previous offset the second location was incorrect. One file contained a value of 0 for the previous pointer which indicates that it had incorrectly written the default value, rather than an offset.

The xref table is not well formatted

Beyond the xref pointer issues seen in the sample set, the table structure itself can be malformed in unexpected ways.

The following examples were reported as Github issues for PdfPig.

No linebreak after xref, for example:

xref5 2
0000000000 65535 f 
0000134883 00000 n

More object entries in a subsection than declared in the header, for example if only 2 objects are declared the table can contain more:

xref
0 2
0000000000 65535 f 
0000000230 00000 n 
0000000520 00000 n 
0000001000 00000 n

Garbage in the middle of the table, for example:

xref
0 2
0000000000 65535 f
0000455.8483a a010 00000 n

Conclusion

We looked at how parsing a PDF should proceed according to the specification. We then compared this with a survey of sample files where we saw a 0.5% error rate due to non-compliant files. All tested PDF viewers (PDF.js, Adobe, Sumatra) were able to open these files because most parsers are extended to support non-compliant files.

This serves as a brief survey of the challenges of parsing a single part of the PDF specification (22 pages out of 1,300 total from version 1.7).

Laureles, Medellin swimming pools

Eliot (Eliot Jones) — Wed, 23 Apr 2025 00:00:00 GMT

Just a brief note to summarise the (non-hotel) swimming pool situation in the Laureles area of Medellin since it is not easy to find/understand all this information.

There are 3 primary organisation types that are relevant:

Inder. Run by the mayorality this sports department offers access to many different sports facilities for residents and visitors alike. Swimming can be accessed for free through Inder.
Liga de Natacion de Antioquia. The swimming league of the department of Antioquia they share access to many of the same facilities with Inder; however they provide paid access to these facilities.
Private organisations. A third category that runs some pools.

It's worth noting that for all pool types the required swimwear is a swimming cap (even for bald people) and lycra (not board) shorts. There are 2 shops at Estadio that sell these, one near the Inder entrance and the other behind the stairs at the Liga entrance. There may also be a shop at Belen.

Pools

The main pools of interest are the Estadio and Belen sites. Belen has a single Olympic pool (see the private swimming section for details on a second 25m pool in Belen) and Estadio has a range of pools including Olympic, sub-aquatic and many others.

Estadio, has an Inder gate with ticket office here and a Liga gate here.
Belen, the pool is here.

Swimming with Inder

To use Inder pools with a reservation you will need to register with Simon 2.0. Simon 2.0 is the booking platform for various sports facilities including the pools at Estadio and Belen. Once registered you can book access to a pool session for free, however as of Spring 2025 there is high demand and because many pools, including the Olympic pool, at Estadio are currently being refurbished the available reservation slots fill up quickly.

Another approach is to join the queue for entry at the pool in Estadio. Each hour (or outside the hour at the discretion of the ticket office) a certain number of people are let in from the queue for free, again depending on occupancy. I don't have first-hand experience with it but in the week it apparently doesn't take too long if you go early. The non-reservation entry queue may also exist at Belen but I haven't validated this.

You will need your original id, either passport or Cedula (no photocopies), to enter with Inder.

Swimming with the Liga

The Liga de Natacion de Antioquia also operates at the Estadio and Belen locations. They don't seem to be mentioned if you ask about swimming in general at Inder locations.

At Estadio you can enter the complex by the 'secret' Liga gate (Accesso #2) and go past the ticket offices and left, round the back to a ticket office where you can pay for individual entry between 9am - 3pm (~COP17.9k). However during the renovations it is hit-or-miss if they can allow this because of the reduced pool capacity. You can ask at the main ticket offices at this gate first to find out if it is possible, I have had varying success depending on the day and hour.

The Liga also shares the Olympic pool in Belen with Inder. However you cannot pay for individual entry here, instead you must purchase an annual membership (~COP85k) and then purchase either a 10-swim or monthly (mensualidad) ticket. If you are in Medellin for a while this might be the most convenient approach since the hours for the Belen pool are longer and you don't have to reserve or take the chance in the queue. As of Spring 2025 the Liga hours for Belen are reported as:

Monday, Wednesday, Friday: 6.30am - noon, 1pm - 4pm
Tuesday, Thursday: 6am - noon, 1 - 4pm
Saturday, Sunday: 7am - 2.45pm (subject to availability)

The Liga ticket office at Belen is also not the easiest to find. You need to walk through the door and past the Inder front-desk and to the left to get to it and this is not obvious from outside. If you ask directly the Inder desk can point you to it.

You will need your original id, either passport or Cedula (no photocopies), to enter with the Liga.

Private swimming

At Belen there is a second pool, the Piscina Corporación Deportiva y Cultural, which is operated by a private company. This is quite a shallow pool 1.1 of metres but has good changing facilities and a secure place to leave your bags. This only costs COP12k for a single ticket with no reservations necessary. Note that there are times when the pool is reserved for classes so you need to confirm the hours beforehand. As of Spring 2025 these are:

Monday - Friday: 10am - noon, 1pm - 4pm
Saturday: 11am - 3pm
Sunday: 11am - 2pm

No id is needed for this pool.

Also worth a mention is the Diana Espinosa Aquatics Center which are smaller pools offering classes for learners/aquafit. These are indoor heated pools.

There may be more pools in this category but these are the ones I am aware of.

Writing Code for Fun and ... That's It

Eliot (Eliot Jones) — Sun, 30 Jul 2023 00:00:00 GMT

Supreme Commander: Forged Alliance is a real-time strategy (RTS) game released in 2007. It is also the last good RTS, and potentially game, ever.

Despite this most gamers — not realizing gaming reached a pinnacle in 2007 and has since descended into a mess of RPGification, Loot Boxes and bloom over-use — have moved on to other games and more importantly other genres¹.

One need only look at the modern-game design horrors unleashed on the Dawn of War genre to see the decline of RTS and gaming generally more clearly. From the high-point of Dawn of War: Winter Assault in 2005 to the best-forgotten mess of Dawn of War III in 2017 everything has been ruined irrevocably. Nothing is good anymore, food doesn't taste the same, music isn't what it was². I'm not getting old, things are getting worse. I'm still cool and relevant!

Few things in gaming quite match the state of stressed, overwhelmed, misery induced by Forged Alliance. Thankfully the Forged Alliance Forever (FAF) project has continued to develop Forged Alliance, providing graphical updates, balance changes, matchmaking, performance improvements, new maps and more.

Because gamers have departed from the True Path matchmaking generally takes a while. A small active user-base means that depending on the time of day and day of week you can be waiting anywhere from 10 minutes to over an hour for a lobby (the pre-game matchmaking bit) to fill.

It is generally necessary to be ready to balance and start the lobby within a few minutes of it filling. If there's too much of a delay everyone will leave again and the waiting time will have been wasted.

Which brings me to the justification for my project, FAF Lobby Sim. While the lobby is running on my desktop I don't want to have to sit checking it every few minutes. It would be good to be able to do other things elsewhere while having a way to check on the occupancy status.

Design Log

What follows is the 'design' process I followed. As you can tell from reading it I really wasn't worrying too much about doing it right, just doing the bare minimum. Developing this way was fun, a nice break from proper open-source and job related development. I wanted to stay in C# .NET because it is the language I know best.

Whatever is stopping you starting on some project I hope this transparency helps encourage you to just churn out some code and get something shipped.

Getting started

So the lobby application is running on my computer and I want to check its status on my phone. The lobby for FAF involves 2 running processes. The Forged Alliance game application itself and the Java FAF client application.

The Java client consumes an API that can report the active lobbies and the occupancy state. This seems a natural starting point. If I can call the same APIs from a mobile client then I'll be done. There are 2 drawbacks to this though:

The API uses OAuth tokens for authentication. I'd need to get my own client id to call it. This is the sort of un-fun development work that involves coordinating between people and waiting for things to get done. Fine for work, not for a weekend project.
While the occupancy reported to the client by the API is generally correct it doesn't match the real state when open slots are manually marked 'closed' from within the game process. This can happen if a map has 12 slots but 2 of them would have insufficient resources. This is a rarity but combined with point 1 made me consider a different approach.

I hatched a Heath Robinson style plan. If I could run some client monitoring application on my desktop, pass the information to some server and then interact with that server from mobile I'd own all the hops. Should be easy right?

What's a plan without some arbitrary constraints? Since the client app will be calling a server I want to:

Make the client app small, with minimal dependencies.
Not pass anything other than the bare minimum information to the server.

Building the client app

So I've done some work with the FlaUI automation library in the past. Here's a FizzBuzz implementation using the Windows 10 calculator.

If I can use FlaUI and grab the information I need from the game lobby process it should be fairly simple. Just poll the game in a loop and report the information to the server. This will address the second drawback listed for using the API directly, since I'm reporting the actual game-state.

So I launch up the FlaUI inspector tool and... there's nothing there, just the window of the process. This is to be expected, it's a game after all, not a UI. The Java client process is far more amenable to UI automation, but it doesn't solve the real-state/closed slots problem I've now fixated on solving.

I vaguely recall a former boss once used OpenCV to 'bot' a simple game. I guess I can do something similar here. Since FlaUI can take screenshots of the open desktop it should be possible to parse out the info I need using image processing. I definitely don't want to send the screenshots to the server process since they will capture the entire desktop, not just the monitored process if other windows are moved in front of the monitored application.

Image processing

This screenshot shows the layout we need to parse. We 'just' need to pull out the number of "Open", "Closed" and occupied slots in the Nickname column.

Obviously we don't want to hardcode the positions since the game can run windowed or full-screen on different resolutions. The number of slots can vary from 2 - 16.

The obvious approach is to use OCR, e.g. Tesseract, to process the text and calculate occupancy from the Nickname column. Pulling in Tesseract means that you have a large dependency for the client application which isn't great if you want to redistribute it. Like everything it's much harder to get Tesseract working on Windows where this client app would be running.

I also played around with OpenCV and edge detection but didn't want the dependency or to read the documentation to figure out what I needed to do. So I settled on the questionable strategy of building the processing myself.

The first step is to threshold the image to simplify the input data. Playing around with the image in GIMP I found a reasonable threshold value:

Applying a threshold to the screenshot was a simple one-liner with ImageSharp:

image.Mutate(x => x.BinaryThreshold(0.39f));

I could then map the thresholded image to a C# BitArray to make it more memory efficient to work with. This would involve flattening the 2D image to an array so it was necessary to have a couple of methods to convert to/from 2D to 1D space:

private static int XyToFlat(int x, int y, WidthHeight widthHeight)
        => (y * widthHeight.Width) + x;

private static (int x, int y) FlatToXy(int index, WidthHeight widthHeight)
{
    var y = index / (widthHeight.Width);
    var x = index - (y * widthHeight.Width);
    return (x, y);
}

With these methods mapping a black-and-white image to a bit array is simple:

private static BitArray FlattenThresholded(Image<Rgb24> image, WidthHeight wh)
{
    var result = new BitArray(image.Height * image.Width);

    for (int row = 0; row < image.Height; row++)
    {
        for (int col = 0; col < image.Width; col++)
        {
            var flatIndex = XyToFlat(col, row, wh);
            if (image[col, row].R == 255)
            {
                result[flatIndex] = true;
            }
        }
    }

    return result;
}

So now we have our image in a black-and-white array (0 and 1). What next?

Well looking at the example thresholded image we can see that the region of interest is the table highlighted in orange:

If the slot is open there are 2 'words' in the Nickname column, "Open" and the chevron symbol for the dropdown containing the option to move into the slot or, if you are the host, close the slot. When I say 'word' here I mean nothing more group of pixels since this can be actual text, a faction symbol, a country flag, a block of color, or something else.

If the slot is closed there is only the word "Closed" and no chevron.

For slots occupied by a player there's either the player name and a chevron or, for the current user, simply the username.

Without OCR to distinguish between "Closed", "Open" and the current username you also need to use the "R" (Rank), "G" (Game count) or "Team" column to identify that a player occupies the current slot (the 'word' in the "Color" column may disappear due to thresholding).

The approach becomes:

Locate the table and columns.
If the Nickname column contains 1 word and no entries in other columns it is closed.
If the Nickname column contains 1 word and there are entries in other columns it is occupied by the current player (assuming a single word nickname)
If the Nickname contains 2 (or more) words and no entries in other columns it is open.
If the Nickname contains 2 (or more) words and one of the other columns (excluding "--") is occupied then the slot is occupied

To locate the table without hardcoding the position we need to first detect 'words'. As mentioned above these might be non-word groups of pixels but for our purposes since we're not running OCR the distinction is meaningless.

How do we detect words? Let's take a look at the "Nickname" text from the thresholded image:

If you're scanning the image top-to-bottom left-to-right when you first encounter a white pixel (marked in green in the image below) you can start the following process:

Look around the neighboring pixels (ignoring pixels to the top-left since we scanned from that location) in a 9-by-9 region.
Each time you find a white pixel in the neighborhood then move to that pixel (new green pixel), recording where you came from (marked blue in the image).
Scan the neighborhood of the new pixel and move to any non-visited white pixels in that location (the new green pixel).

The first steps of this are shown below, the colors are purely illustrative:

This is basically just a flood fill algorithm except you allow for gaps, both for diagonal pixels and entirely disjoint pixels like the dot on the 'i' and next letters in the word.

The problem with this approach is I wrote it recursively and it causes a stack overflow if the region to be filled is too large. Since large regions are just background noise in the thresholded image I just exit the recursion early if the region to be filled is too large, for some arbitrary value of too large. I should probably have implemented a proper flood fill algorithm instead.

Full code for this is here.

Using this flood fill word detection the resulting words are fairly well detected and defined:

Once the 'words' are detected there are just some fairly gnarly heuristics to group them into lines and identify the table header and columns.

Building the server app

The client runs in a loop, every 5 seconds it uses FlaUI to take a screenshot of the lobby process, run word detection and use the heuristics discussed earlier to calculate the total slots and number occupied.

The server part is an incredibly simple MVC app that just holds the data in memory. The client posts to the controller which stores the latest occupancy and sends a server-sent event to any clients monitoring the lobby.

The idea is each lobby will be assigned a random identifier by the client on startup and multiple clients and lobbies may report occupancy to the server based on the identifier. In order just to ship it I hardcoded the lobby to a single identifier at this stage.

The final product is not great looking, but does work, and seems to update fairly reliably:

Conclusion

Really at this point I couldn't be bothered to put in the necessary work to 'productionize' it. To make a real usable implementation I'd need to:

Address the test case that is failing due to thresholding in ImageSharp not matching GIMP
Make the heuristics for occupancy detection less flaky
Package the client application and make it more reliable
Improve the web UI and add rate limiting and other DDOS protection
Dynamically generate lobby ids and assign a random token to each lobby id so only the client that first reports values for the lobby can send updates for it

With all those to-dos in mind I'm pleased I managed to get anything shipped and the few times I've used it it has meant I can get live updates pushed to my phone while being away from the computer.

The code for the client app is super funky and not at all idiomatic C#. I'm hoping to write a post about why I wrote it that way and what I learned from it soon.

Footnotes

An honorary mention to Beyond All Reason an open-source spiritual successor to Supreme Commander.
I don't believe this at all.

Blog Update

Eliot (Eliot Jones) — Sat, 29 Jul 2023 00:00:00 GMT

As this blog entered its tenth year it was desperately in need of a facelift. I had effectively stopped writing new posts because the overhead of remembering how to upload a new post each time was off-putting enough to stop me entirely.

For that reason it has been updated with new - hopefully more consistent - CSS. In addition the code and posts are deployed on each commit by GitHub actions. This means I no longer need to SSH into the server to do things by hand. This should mean the blog is editable from wherever I have git access. I also removed the terribly insecure previous method of uploading posts directly onto the server which was using a plain-text username and password.

Most importantly privacy-invading Google Analytics has been removed along with almost all JavaScript. I no longer see the need for analytics on this blog. The only remaining scripts are for code highlighting client-side as well as Disqus for commenting. Disqus is opt-in so will only load in if you click the "Load Comments" button at the bottom of each post.

Unfortunately I still rely on Google Fonts for the title font but the main body font has been updated to use the system font stack.

Finally it has been updated to .NET 7. The previous version was running .NET Core 1.1 and hadn't been updated since 2017. So much for keeping software up-to-date! The new version is on .NET 7 and the upgrade process was surprisingly easy.

While the changes between the ASP .NET versions have been a little hard to keep track of with Startup.cs being removed and everything moving into Program.cs the migration was made simple by my lazy cheat. I just created a new ASP .NET 7 application from the template in Visual Studio and moved most of the files without any changes, except to update the namespaces.

I also took the opportunity to add caching. Previously it loaded every post file from disk repeatedly just to show a single page. Now, because posts are only updated whenever the app itself is deployed and restarted, cache invalidation is trivial.

I have added the images into git too which generally causes a lot of squawking from people who use git properly (nerds), but keeping things simple should hopefully mean I fall into the pit of success. Each PNG image has been compressed further where possible.

Deployment

Deployment of the blog is pretty much unchanged from the initial set-up following the guide by Scott Hanselman for deploying .NET to a Linux server such as a DigitalOcean VPS. This may no longer be best-practice but has been working for me uninterrupted for over half a decade.

My DigitalOcean VPS is the cheapest tier ($5 a month originally, now $6-something thanks to inflation) and hosts this blog plus a couple of other applications.

The application is published from either my Windows machine or the GitHub CI runner using:

dotnet publish -c Release -r linux-x64

Which generates a self-contained application (when <SelfContained> is defined in the .csproj file). The application is then copied to the server (not directly to the deployed directory since this is locked by the running app) using scp:

scp -r /c/path/to/publish username@ip-or-host:/path/on/remote

Each application is deployed to the /var/{app-folder} directory on the server and launched and managed by Supervisor.

The Supervisor config files live at /etc/supervisor/conf.d. Each file defines an application for Supervisor to manage. This blog lives under /var/blog:

[program:blog]
command=/usr/bin/dotnet /var/blog/LightBlog.Server.dll --server.urls:http://*:5000
directory=/var/blog
autostart=true
autorestart=true
stderr_logfile=/var/log/blog.err.log
stdout_logfile=/var/log/blog.out.log
environment=ASPNETCORE_ENVIRONMENT=Production
user=www-data
stopsignal=INT

To copy the newly scp published app to the /var/{app-folder} directory the running application (named blog) is first stopped using:

supervisorctl stop blog

The existing files are then cleared, the newly uploaded files copied to the target folder and finally the blog restarted using:

supervisorctl start blog

There are more details in Scott's blog-post for configuring NGINX as a reverse proxy and using Let's Encrypt to expose the site over HTTPS.

The GitHub actions workflow for deployments is here.

Well that's all for this update. I'm hoping to write a post about building stupid code for fun to explain FAF Lobby Sim (warning, not HTTPS!) soon, but based on my track record see you in 2030...

Visual Studio 2022 Debugger Freezes

Eliot (Eliot Jones) — Fri, 18 Feb 2022 00:00:00 GMT

Wow, blogging with any kind of regularity is hard I kind of hard I guess. Who knew?

Anyway this was just a quick note for a problem I had recently.

I found that my Visual Studio 2022 debugger would hang, freeze or become unresponsive intermittently. Prior to the last update the process would freeze completely, however even after updating it would freeze though the IDE itself would remain responsive.

This freeze seemed to trigger more frequently when using time-travel debugging.

As usual when weird things happen the antivirus was to blame.

Excluding the process named VsDebugConsole.exe seemed to resolve at least all the debugger hangs after adding the exclusion. No doubt some more will occur but at least VS22 is usable again.

To get to exclusions on Windows 10 (old-skool) go to Windows Security -> Virus & threat protection -> Virus & threat protection settings -> Manage settings -> Exclusions -> Add or remove exclusions.

Attention Bubbles (Or Why Everything Happens So Much)

Eliot (Eliot Jones) — Fri, 31 Jan 2020 00:00:00 GMT

Like most side projects that devour hours of time and end up being entirely useless, this one started with a simple question:

Can recessions be predicted by increased interest in news stories about recessions?

The background behind this question -- and you'll have to forgive my complete ignorance of basically the entire field of economics -- is that recessions are a feature of capitalist economies. From what little I've read (and understood); during a boom, businesses scale up their operations to produce more and more "stuff". At a certain point the amount of "stuff" exceeds the demand for "stuff" and businesses and sectors are left holding a load of plant/equipment/employees they no longer need.

From this (probably wrong) summary of boom and bust it seems there's a certain extent to which recessions are based on people getting scared. When a consumer/business has lower confidence they're less likely to buy a new thing or make a plan for something, thereby reducing demand. This has a cascade effect throughout the market and causes a crash. It's basically a mass-panic. (I appreciate this description is probably making a lot of economists very angry, but I'll counter with my (tongue-in-cheek) belief that neither economics nor money are real, they're just belief systems.)

Given the mass-panic and social nature of recessions we should be able to spot the signs of a panic slightly before they happen. In the same way that you could tell that TikTok was going to be the next big thing -- yes this is probably going to date this article horribly by the end of the month -- by looking in the right places for increased interest in the app, signs of decreasing confidence should appear somewhat before the main event.

For this analysis I turned to what I affectionately call the "Orange Hell Site". Hacker News is a site for news stories across a wide range of topics, mainly focused on topics of interest to Silicon Valley; technology, economics, science and Venture Capitalist culture. Because of its Silicon Valley focus it tends to broadly reflect the beliefs of the elite in Silicon Valley; whether that's techno-utopian solutions to the climate crisis or the belief that someone with no qualifications working 2 minimum wage jobs can reach FIRE (Financial Independence Retire Early) just by applying themselves. This also means it takes a keen interest in all things financial markets, stocks, shares and whatever-the-hell RSUs are.

TLDR: by looking at what people with a lot of involvement and interest in the stock market are talking about we're going to see if recessions might have signals preceding them.

One of the main problems with this approach is that fortunately for everyone but unfortunately for the analysis, recessions don't happen often. The last Big One was 2007/8, it seems likely we're due another one anytime soon, but by their very nature that's impossible to predict. Hacker News launched in February of 2007, just before the last recession so we have a very limited set of data to look at.

Building It

The first problem I encountered was the nature of the Hacker News API. While it was extremely useful to have any API, the design of the API requires enumerating every single story, comment, job posting and Ask HN by ID just to find the stories.

I excluded comments from the analysis since they didn't really seem relevant. I was more interested in the higher-threshold activity of submissions since I felt they would more accurately reflect what was being seen by people.

At the time the question came to mind I was observing what I believed to be a bubble in stories about recessions (August 2019). In order to validate the approach without enumerating all 22 million items (and flooding the API with requests) I initially downloaded the data with IDs from about 16 million to about 20 million (2017 - 2019).

The initial dataset was, frankly, useless:

It did appear to show something like an increase in frequency of stories about recession though. I just needed a lot more data.

This resulted in the need to enumerate the other 16 million entries from 0 - 16,000,000 as well as the new entries from 20 million - 22 million. The crawler I used for the first data-set was very slow. It was using a single thread and a delay between requesting each item (in an attempt to be a good citizen/API consumer) it had taken weeks of running the crawler overnight just to fill the initial 4 million set.

Since I was running everything on my laptop and didn't want to pay anything to solve what was ultimately a stupid question, any solution was constrained by the lack of disk space, CPU and memory on my fairly cruddy laptop.

The crawler went through many iterations before reaching its final form. The final crawler is a C# .NET Core console application which uses a configurable number of threads and writes its data to a SQLite backing store. This SQLite database is shared by all applications/ Initially I used my own binary format to reduce file-size but this wasn't very useful when it came to running the crawler and querying the data from a web application.

I also needed a way to view the data. There's probably a way to analyse the data with statistical methods but at this point my motivation changed from building some kind of predictive model (nowhere near enough data) to building something to visually inspect whether any topic, not just recessions, was undergoing what I call an "attention bubble". To this end I build an ASP.NET Core website.

As it turned out, perhaps unsurprisingly, running a `WHERE title LIKE '%recession%" search on SQLite is very slow. This was a good opportunity to learn a new thing, so I added a Lucene.NET index to the mix for free text search. The indexer code is here, as it was my first time using Lucene.NET I think I probably missed a whole load of potential optimizations so it's still fairly slow, but much better than it was when using SQLite and probably acceptable.

The end result is hosted here. It provides search capabilities for terms in story titles in Hacker News postings. Due to the aforementioned "not wanting to spend money on a stupid question", it's piggybacking on this blog's VPS (which is the lowest tier possible machine) so will fall over above about 3 concurrent users, so go easy on it. It runs an update every hour to crawl new stories and index them using ASP.NET Core's BackgroundService. This should probably be hived off to a separate service but having it in-process makes cache invalidation a lot easier, though there's no real drawback to serving stale data for something like this. Also if you hit the site after a cache invalidation it's going to be super-slow for the first request.

A disclaimer on the front-end code: I built the front-end as a means to an end (no pun intended). For this I used jQuery and didn't even bother retrieving the data from an API endpoint. Instead the data gets bundled up with the server HTML response. For this reason the front-end is a garbage fire.

Pretty Pictures

Enough talking, let's look at some of the data. First up, our (revised) initial question, was I observing an 'attention bubble' about recessions? This is the count of stories with "recession" in the title by month:

Since there are more posts per-day nowadays it's also useful to look at the count as a percentage of all posts in the period. This is the same data as a percentage of stories in the period rather than count:

Circled in blue is the last "Big One", 2007/8 but it has a long tail into 2009/10. In red is the peak I was observing in Fall/Autumn of 2019. It seems to have fizzled out, perhaps QE is keeping the economy afloat for a few more months, but it will be interesting to see how it develops. There was definitely a spike but whether it represents a foreshock is unknowable.

An example of a pure attention bubble might be the "Cybertruck", Tesla's new electric truck announcement generated a lot of buzz, with people split between it being the worst vehicle ever designed or a design masterpiece:

The big spike around the announcement seems to have fizzled out, again it will be interesting to see how it develops, if the Cybertruck is ever actually released.

Another topic that is unfortunately in the news is the, as yet unnamed, coronavirus. The SARS crisis of 2003 was also a coronavirus which sadly killed quite a few people but was luckily contained, partly because people became symptomatic prior to being infectious; so people tended to have been contained prior to spreading the virus. (A great read on the topic of viruses including SARS as well as other zoonoses is Spillover by David Quammen). Hopefully this coronavirus will also be easily contained. Again we see a huge spike of interest in this topic at the right of the graph, which will hopefully fizzle out quickly. Also circled in red are the only mentions of coronavirus in submissions prior to 2020:

On a less sombre note we can also look at how named technologies are trending, and how they trend after launch, for example Kubernetes as a % by month:

And the trend of another technology in the same field, Docker:

Next Steps

I don't really intend to do much more work on this project, the result serves its purpose and I've got PdfPig to keep me occupied. However some enhancements would be:

Improve scoring of search terms in Lucene using boosting for certain fields.
Improve performance. Other than putting it on a bigger machine it remains to be seen what the performance is like under high load. It's likely upgrading to .NET Core 3.0 from 2.1 would help here, there are a lot of performance improvements to the framework between these versions. It also seems like sharding the Lucene index might help but that's a lot of work for a side-project.
Allow multiple trends on the same plot. It would be interesting to compare Kubernetes with Docker directly, or PyTorch with Tensorflow to get an idea of the relative trends in attention.
Accounting for score received by the submission on Hacker News. This would probably be the most important metric to account for and in retrospect was the main thing I would have changed if I were starting building this now, storing a snapshot of the score -- perhaps after a lead time of several days -- would allow more interesting analysis of the popularity of the topics outside of a naive count.

Bonus round, TikTok trend:

Bezier Curve Bounding Boxes

Eliot (Eliot Jones) — Wed, 18 Dec 2019 00:00:00 GMT

One of the challenges for generating accurate character sizes in a PDF document I encountered while building PdfPig was working out the bounding box for a cubic Bezier curve.

A cubic Bezier curve is defined by 4 points; the start, end and 2 control points.

We can number the points for use in the formulae in this post:

Start: P0
Control 1: P1
Control 2: P2
End: P3

This gives the formula for the Bezier curve:

We can take advantage of the property that the gradient of the curve will be 0 at the bounds of the curve (as well as including the start and end of the curve).

To find where the gradient is 0 we can differentiate the curve and then solve the differentiated equation for 0.

The formula differentiated (from Wikipedia) is:

Where P0 is the start control point with both x and y values. Taking just the X dimension (and moving the (1 - t)), we rewrite the formula as:

P'(x) = 3(P1.x - P0.x)(1 - t)^2 + 6(P2.x - P1.x)(1 - t)t + 3(P3.x - P2.x)t^2

Next we can assign some names to some known values:

var i = (P1.x - P0.x)
var j = (P2.x - P1.x)
var k = (P3.x - P2.x)

Allowing us to rewrite the formula as:

P'(x) = 3i(1 - t)^2 + 6j(1 - t)t + 3kt^2

Now let's multiply out the full thing:

P'(x) = 3i - 3it - 3it + 3it^2 + 6jt - 6jt^2 + 3kt^2

And tidy it up a bit:

P'(x) = 3i - 6it + 6jt + 3it^2 - 6jt^2 + 3kt^2
P'(x) = 3i + (-6i + 6j)t + (3i - 6j + 3k)t^2

Given we are finding the point where the gradient is 0:

3i + (-6i + 6j)t + (3i - 6j + 3k)t^2 = 0

If we reorder the terms we see this is a quadratic equation in disguise:

(3i - 6j + 3k)t^2 + (-6i + 6j)t + 3i = 0

The quadratic equation given by:

ax^2 + bx + c = 0

Means we need the following values:

a = (3i - 6j + 3k);
b = (6j - 6i);
c = 3i;

This is just a quadratic equation which we can solve using the quadratic formula to find the value of t:

t = -b (+/-) sqrt(b^2 - 4ac)) / 2a

We will get between 0 and 2 solutions for t depending on if:

b^2 - 4ac

Is greater than zero. In addition the start and end control points provide us with the other points to build our bounding box.

How might this look in code? My full implementation is here which is used to calculate the bounding box of Bezier curves in PDF documents and Type 1 fonts.

To start with let's define our points:

Point start = new Point(5, 7);
Point end = new Point(10, 9);
Point control1 = new Point(6, 5);
Point control2 = new Point(9, 12);

Now taking just the x dimension and the naming we established above:

double p0 = start.X;
double p1 = control1.X;
double p2 = control2.X;
double p3 = end.X;

Now we can substitute our named values (i, j and k):

double i = p1 - p0;
double j = p2 - p1;
double k = p3 - p2;

Now for the parts of the quadratic equation (a, b and c) we established above:

double a = (3i - 6j + 3k);
double b = (6j - 6i);
double c = 3i;

First we need to check if there are any real numbers that solve the sqrt(b^2 - 4ac) part of the quadratic equation. If the result of b^2 - 4ac is negative there are only imaginary solutions to the problem.

double sqrtPart = (b * b) - (4 * a * c);
bool hasSolution = sqrtPart >= 0;
if (!hasSolution) return;

If we have real solutions to the square root we can then evaluate the solutions for the value of t:

double t1 = (-b + Math.Sqrt(sqrtPart)) / (2 * a);
double t2 = (-b - Math.Sqrt(sqrtPart)) / (2 * a);

For Bezier curves, values of t are only valid if they lie between 0 and 1, so if we have values in this range we can substitute them into the original Bezier equation to find the minimum and maximum values of x:

P = (1 - t)^3(P0) + 3(1 - t)^2(t)(P1) 
    + 3(1 - t)(t^2)(P2) + (t^3)P3

So in code:

public static double GetSolutionForT(double t, double p0, double p1, double p2, double p3)
{
    double oneMinusT = (1 - t);

    return (Math.Pow(oneMinusT, 3) * p0)
        + (3 * Math.Pow(oneMinusT, 2) * t * p1)
        + (3 * oneMinusT * Math.Pow(t, 2) * p2)
        + (Math.Pow(t, 3) * p3);
}

This will give 0, 1 or 2 solutions for t (depending on whether t0 and t1 were in the range 0 to 1). The same equation can then be run for y by simply setting p0 = start.Y; and the same for other points.

The full code is:

public struct Point
{
    public double X { get; }
    public double Y { get; }

    public Point(double x, double y)
    {
        X = x;
        Y = y;
    }
}

public struct Rectangle
{
    public Point Min { get; }
    public Point Max { get; }

    public Rectangle(Point min, Point max)
    {
        Min = min;
        Max = max;
    }
}

public static class BezierSolution
{
    public static Rectangle GetBounds(Point start, Point control1, Point control2, Point end)
    {
        (double? solX1, double? solX2) = SolveQuadratic(start.X, control1.X, control2.X, end.X);
        (double? solY1, double? solY2) = SolveQuadratic(start.Y, control1.Y, control2.Y, end.Y);

        var minX = Math.Min(start.X, end.X);
        var maxX = Math.Max(start.X, end.X);

        if (solX1.HasValue)
        {
            minX = Math.Min(minX, solX1.Value);
            maxX = Math.Max(maxX, solX1.Value);
        }

        if (solX2.HasValue)
        {
            minX = Math.Min(minX, solX2.Value);
            maxX = Math.Max(maxX, solX2.Value);
        }

        var minY = Math.Min(start.Y, end.Y);
        var maxY = Math.Max(start.Y, end.Y);

        if (solY1.HasValue)
        {
            minY = Math.Min(minY, solY1.Value);
            maxY = Math.Max(maxY, solY1.Value);
        }

        if (solY2.HasValue)
        {
            minY = Math.Min(minY, solY2.Value);
            maxY = Math.Max(maxY, solY2.Value);
        }

        return new Rectangle(new Point(minX, minY), new Point(maxX, maxY));
    }

    private static (double? solution1, double? solution2) SolveQuadratic(double p0, double p1, double p2, double p3)
    {
        double i = p1 - p0;
        double j = p2 - p1;
        double k = p3 - p2;

        // P'(x) = (3i - 6j + 3k)t^2 + (-6i + 6j)t + 3i
        double a = (3 * i) - (6 * j) + (3 * k);
        double b = (6 * j) - (6 * i);
        double c = (3 * i);

        double sqrtPart = (b * b) - (4 * a * c);
        bool hasSolution = sqrtPart >= 0;
        if (!hasSolution)
        {
            return (null, null);
        }

        double t1 = (-b + Math.Sqrt(sqrtPart)) / (2 * a);
        double t2 = (-b - Math.Sqrt(sqrtPart)) / (2 * a);

        double? s1 = null;
        double? s2 = null;

        if (t1 >= 0 && t1 <= 1)
        {
            s1 = GetBezierValueForT(t1, p0, p1, p2, p3);
        }

        if (t2 >= 0 && t2 <= 1)
        {
            s2 = GetBezierValueForT(t2, p0, p1, p2, p3);
        }

        return (s1, s2);
    }

    private static double GetBezierValueForT(double t, double p0, double p1, double p2, double p3)
    {
        double oneMinusT = 1 - t;

        return (Math.Pow(oneMinusT, 3) * p0)
                + (3 * Math.Pow(oneMinusT, 2) * t * p1)
                + (3 * oneMinusT * Math.Pow(t, 2) * p2)
                + (Math.Pow(t, 3) * p3);
    }
}

Open and Create PNG Images in C#

Eliot (Eliot Jones) — Wed, 18 Dec 2019 00:00:00 GMT

I wanted a platform independent way to open and create PNG images in C#. BigGustave is a new library which provides a .NET Standard 2.0 compatible way of opening and creating PNG images.

To open a png image you can pass either the bytes or the stream of the image to Png.Open and then retrieve the values for pixels at any location in the image:

Png png = Png.Open(File.ReadAllBytes(@"C:\pictures\example.png"));
Pixel first = png.GetPixel(0, 0);
Console.WriteLine($"R: {first.R}, G: {first.G}, B: {first.B}");

To create a .png image in C# use the PngBuilder to define pixel values before saving to an output stream:

var builder = PngBuilder.Create(2, 2, false);

var red = new Pixel(255, 0, 0);

builder.SetPixel(red, 0, 0);
builder.SetPixel(red, 1, 1);

using (var memory = new MemoryStream())
{
    builder.Save(memory);

    return memory.ToArray();
}

BigGustave is completely open source and is available on NuGet now so if you need very basic PNG manipulation tools for platform independent .NET code why not check it out?

PdfPig Version 0.0.5

Eliot (Eliot Jones) — Sun, 30 Dec 2018 00:00:00 GMT

Today version 0.0.5 of PdfPig was released. This is the first version which includes the ability to create PDF documents in C#.

There aren't many fully open source options around for both reading and writing PDF documents so the addition of PDF document creation to PdfPig is an exciting next step for the API.

The actual design of document creation isn't finished yet and there's more work to be done around the currently unsupported use cases such as splitting, merging and editing existing documents as well as adding non-ASCII text, working with forms and adding images to new documents but the functionality in 0.0.5 should provide enough for simple use cases and the open source Apache 2.0 license means that it can be used in commercial software.

You can create a new document using a document builder:

PdfDocumentBuilder builder = new PdfDocumentBuilder();

This creates a completely empty document. To add the first page we use the imaginatively named add page method.

PdfPageBuilder page = builder.AddPage(PageSize.A4);

This supports various page sizes defined by the PageSize enum, such as the North American standard PageSize.Letter. It also allows the choice of portrait (default) or landscape pages.

Once a page builder has been created text, lines and rectangles can be added to it.

Fonts

In order to draw text a font must be chosen. Version 0.0.5 supports TrueType fonts as well as the 14 default fonts detailed in the PDF Specification. These are called the Standard 14 fonts and while their use is beginning to be phased out, all PDF readers should still support them.

A Standard 14 font is the quickest way to get started since it doesn't require a TrueType file. Because the fonts are already well defined and included with PDF reading software the font does not need to be embedded in the PDF file, leading to much smaller file sizes. The available Standard 14 fonts are represented by the Standard14Font enum. They are different flavours of the following fonts:

Times New Roman
Helvetica
Courier
Symbol
Zapf Dingbats

To use a font it should be registered with the document builder. By adding the font to the document builder rather than at the page level, it can be shared across pages in multi-page documents. To use Helvetica:

PdfDocumentBuilder.AddedFont font = builder.AddStandard14Font(Standard14Font.Helvetica);

Or for Times New Roman:

PdfDocumentBuilder.AddedFont font = builder.AddStandard14Font(Standard14Font.TimesRoman);

Text

Once a font is registered with a document builder, all pages in that document can use it to draw text. To write some text on the page created earlier we call the AddText method:

IReadOnlyList<Letter> letters = page.AddText("Hello World!", 12, new PdfPoint(25, 520), font);

This call will write the text "Hello World" in size 12 font at the provided location, 25 units from the left edge of the document and 520 units from the bottom of the document. Remember coordinates in PDF run up from 0 at the bottom rather than having 0 at the top as in WPF or other graphics systems. The last parameter is the font we registered earlier.

This method adds the necessary operations to the PDF content stream for this page and returns the letters that will be drawn on the page and their position. This allows you to measure text and draw underlines, or borders around it.

If you want to just find out the sizes and positions of the letters prior to drawing, for example to calculate text wrapping, you can use the measure method:

IReadOnlyList<Letter> letters = page.MeasureText("Hello World!", 12, new PdfPoint(25, 520), font);

The only difference with this method is that it will not add the text to the page and so does not appear in the output document.

Geometry

In addition to text you can also draw lines and rectangles:

page.DrawLine(new PdfPoint(25, 70), new PdfPoint(100, 70), 3);
page.DrawRectangle(new PdfPoint(30, 200), 250, 100, 0.5m);

This draws a line from (25, 70) to (100, 70) with a line width of 3 units. This means a horizontal line is drawn 70 units from the bottom of the page from 25 units from the left edge to 100 units from the left edge.

The rectangle will be drawn from (30, 200) with a width of 250 units and a height of 100 units with a line width of 0.5 units.

Color

The color of text and geometry can also be set. Stroke color (the color of lines) and Fill/Text color are set by separate methods:

// Set line color in RGB.
page.SetStrokeColor(250, 132, 131);
// Set text color in RGB (and the color of any fill operations, not currently used).
page.SetTextAndFillColor(250, 132, 131);

These colors are expressed in RGB from 0 - 255 for the red, green and blue component. Internally PDF uses values from 0 - 1.0 for RGB, this conversion happens within the page builder.

Once a color is set it will remain active for all future calls to add text, drawing lines and drawing rectangles. To set the color back to the default black you must call:

page.ResetColor();

Which means further calls will use the color black.

Full Example

The simplest "Hello World!" example is:

PdfDocumentBuilder builder = new PdfDocumentBuilder();

PdfPageBuilder page = builder.AddPage(PageSize.A4);

PdfDocumentBuilder.AddedFont font = builder.AddStandard14Font(Standard14Font.Helvetica);

page.AddText("Hello World!", 12, new PdfPoint(25, 520), font);

byte[] result = builder.Build();

Further examples including multiple pages can be found in the tests for the document builder.

TrueType

In addition to Standard 14 fonts TrueType fonts (fonts with the .ttf extension) are also supported. The raw bytes of the font files must be provided to the AddTrueTypeFont method. This example uses the System font Baskerville Old Face on Windows 10:

string file = @"C:\Windows\Fonts\BASKVILL.TTF";

byte[] bytes = File.ReadAllBytes(file);

PdfDocumentBuilder.AddedFont font = builder.AddTrueTypeFont(bytes);

Since the result of adding a TrueType font is an PdfDocumentBuilder.AddedFont which is identical to the result for Standard 14 fonts it can be used in the same way to add and measure text for pages.

The TrueType font is interpreted and a compressed version of the font file is embedded and distributed with the resulting PDF document. Currently there are no optimizations to use a subset of the font file which most other document producers do so the file size for documents will be larger.

Conclusion

While it's still early days for the library the addition of Open Source C# PDF document creation should widen the appeal of the PdfPig library and help people who need a truly open source, fully managed .NET PDF creator.

Sentence Boundary Detection in C#

Eliot (Eliot Jones) — Sun, 30 Sep 2018 00:00:00 GMT

Sentence Boundary Detection or Segmentation is the task of splitting an input passage of text into individual sentences. Since the period '.' character may be used in numbers, ellipses or names it's not enough to simply split by the period character.

When I was researching ways to do this in C# I didn't find much in the way of properly open source libraries. A lot of the libraries I found for other languages referred to the Golden Rule Set (GRS). This set comes from Pragmatic Segmenter, a Ruby gem to segment text based on rules observed from a varied corpus of text.

Since I find porting code from other languages helps me understand both the variations in how different languages approach the same problems and also how other people make architectural decisions and structure their code I decided to port it to C#.

This Pragmatic Segmenter port is available to download from NuGet. The public API is similar to that for the Ruby package however the method is static:

var result = Segmenter.Segment("There it is! I found it.");

Assert.Equal(new[] { "There it is!", "I found it." }, result);

There is also support for other languages, the Language enum gives the supported languages:

var result = Segmenter.Segment("Salve Sig.ra Mengoni! Come sta oggi?", Language.Italian);
Assert.Equal(new[] { "Salve Sig.ra Mengoni!", "Come sta oggi?" }, result);

The source code also contains a set of data from various sources I was using to test my port as well as add some behaviour for the sources I was primarily interested in (academic journals). This data can be found here. Hopefully this corpus of annotated sentence boundary data will be useful to people building their own libraries.

Using ConvNetSharp With Feature Based Data

Eliot (Eliot Jones) — Sat, 21 Apr 2018 00:00:00 GMT

ConvNetSharp which is descended from ConvNetJs is a library which enables you to use Neural Networks in .NET without the need to call out to other languages or services.

ConvNetSharp also has GPU support which makes it a good option for training networks.

Since much of the interest (and as a result the guides) around Neural Networks focuses on their utility in image analysis, it's slightly unclear how to apply these libraries to numeric and categorical features you may be used to using for SVMs or other machine learning methods.

The aim of this blog post is to note how to acheive this.

Let's take the example of some data observed in a scientific experiment. Perhaps we are trying to predict which snails make good racing snails.

Our data set looks like this:

Age   Stalk Height    Shell Diameter    Shell Color   Good Snail?
1     0.52            7.6               Light Brown   No
1.2   0.74            6.75              Brown         Yes
1.16  0.73            7.01              Grey          Yes
etc...

ConvNetSharp uses the concept of Volumes to deal with input and classification data. A Volume is a 4 dimensional shape containing data.

The 4 dimensions are:

Width
Height
Depth
Number of rows/observations

Width and height only make sense when dealing with image data so we set them to 1. For our data the depth is equal to our number of features. Because we need to map our categorical features to numbers we end up with the following features:

Age
Stalk Height
Shell Diameter
Is Shell Brown
Is Shell Light Brown
Is Shell Grey

So the depth of our input volume is 6.

Let's create our Net first:

Net<double> net = new ConvNetSharp.Core.Net<double>();

Now we define the input layer based on the shape of the data we will provide to it:

// Define the width and height (1) and the depth equal to our number of features
net.AddLayer(new InputLayer(1, 1, 6));

Then we define some layers:

net.AddLayer(new FullyConnLayer(32));
net.AddLayer(new ReluLayer());

// ConvNetSharp supports dropout layers.
// net.AddLayer(new DropoutLayer(0.3));

net.AddLayer(new FullyConnLayer(32));
net.AddLayer(new ReluLayer());

// The last layer before the Softmax for classification must have as many neurons as output features
net.AddLayer(new FullyConnLayer(2));
net.AddLayer(new SoftmaxLayer(2));

Our last layer is a SoftMax layer which will predict 2 classes, false and true.

To provide our data to train the network we need to take the following steps:

(Optionally) Resample the data if one output is strongly represented in the training data to balance out the classes.
Split the data into a training and test set, the trained network will be evaluated against the test set which is not provided to it during training.
Normalize the data. Store the normalization parameters applied to the training set and apply them against the test set.

First we create a trainer for our network, we can either use Stochastic Gradient Descent (SGD) or an Adam trainer.

We will use SGD for this demo:

var trainer = new ConvNetSharp.Core.Training.Double.SgdTrainer()
{
  LearningRate = 0.01,
  L2Decay = 0.001,
  BatchSize = 70
};

Once we have normalzed our training data we must convert it to a volume to feed into the trainer.

If our training set has 70 rows we have input data of the following form - double[70][6] - where each row is an array containing the numeric value of each of the 6 features.

Our output data is then - bool[70] - where each entry indicates the result, whether or not the snail is good at racing, for each row.

To convert the input data we need our volume to have width and height of 1, a depth of 6 (the number of features) and a 4th dimension equal to the number of rows in the dataset.

Prior to this we need to flatten our input into a single array of the form - double[70 * 6]. To do this each row should be mapped to a consecutive set of 6 entries in the array. For example:

var featuresPerRow = input[0].Length;

var flattenedInput = new double[input.Length * featuresPerRow];
for (int i = 0; i < input.Length; i++)
{
    var row = input[i];
    for (int j = 0; j < row.Length; j++)
    {
      flattenedInput[(i * featuresPerRow) + j] = row[j];
    }
}

This takes each row in the input and maps it to a run of 6 numbers in the flattened array.

Now we can convert this flattened array to a volume:

var volume = BuilderInstance.Volume.From(flattenedInput, new Shape(1, 1, 6, 70));

Where 70 is whatever the number of rows in the input data was and 6 is the number of features in each row.

The output volume is then a boolean array flattened to 2 numbers per boolean:

var flattenedOutput = new double[output.Length * 2];
for (int i = 0; i < output.Length; i++)
{
  var value = output[i];
  flattenedOutput[2 * i] = value ? 0 : 1;
  flattenedOutput[(2 * i) + 1] = value ? 1 : 0;
}

var outputVolume = BuilderInstance.Volume.From(flattenedOutput, 
new Shape(1, 1, 2, output.Length));

This creates an array where each pair of numbers indicates either false (0, 1) or true(1, 0).

To create our test volume:

double[][] testSet;
// Apply whatever normalization technique you are using, e.g. min-max
double[][] normalizedTestSet = Normalize(testSet, trainingSetParameters);

// Apply the same flattening process as we did for our training input array
double[] flattenedTestSet = Flatten(normalizedTestSet);

var testVolume = BuilderInstance.Volume.From(flattenedTestSet, 
new Shape(1, 1, 6, normalizedTestSet.Length));

Then we need to run the training process multiple times (epochs) until we start to converge on an accuracy for our test set:

for (int i = 0; i < maxEpochs; i++)
{
  trainer.Train(volume, outputVolume);

  // Evaluate the test set accuracy
  var predictions = net.Forward(testVolume);

  var numberWrong = 0;
  for (int j = 0; j < testOutput.Length; j++)
  {
    var predictedFalseValue = predictions.Get(0, 0, 0, j);

    var predictedValue = predictedFalseValue < 0.5 ? true : false;

    if (predictedValue != testOutput[j])
    {
      numberWrong++;
    }
  }

  var percentIncorrect = (numberWrong /(double) testOutput.Length) * 100;
}

If you've used a dropout layer in the net remember to disable it before evaluating the test set.

This is the most basic way of training the network. You can enhance this by:

Splitting the training of each epoch into steps by partitioning your input set into batches of, for example, 20 rows.
Cross validating by switching the data used in the test and training data multiple times.
Adjusting or automatically tuning training parameters or network shape/number of layers.

If the output you want to predict is a continuous value rather than a classification you need to train a regression. To do this you swap the last SoftMax layer with a Regression layer. For details on this see this example.

Alpha Release of PdfPig

Eliot (Eliot Jones) — Wed, 17 Jan 2018 00:00:00 GMT

I'm very pleased to finally have reached the first alpha release of PdfPig (NuGet).

PdfPig (GitHub) is a library that reads text content from PDFs in C#. This will help users extract and index text from a PDF file using C#.

The current version of the library provides access to the text and text positions in PDF documents.

Motivation

The library began as an effort to port PDFBox from Java to C# in order to provide a native open-source solution for reading PDFs with C#. PdfPig is Apache 2.0 licensed and therefore avoids questionably (i.e. not at all) 'open-source' copyleft viral licenses.

I had been using the PDFBox library through IKVM and started the project to investigate the effort required to make the PDFBox work natively with C#.

In order to understand the specification better I rewrote quite a few parts of the code resulting in many more bugs and fewer features than the original code.

As the alpha is (hopefully) used and issues are reported I will refine the initial public API. I can't forsee the API expanding much beyond its current surface area for the first proper release.

Usage

To get the text from a PDF using C# with PdfPig the following code opens and retrieves a page from the document:

using UglyToad.PdfPig;
using UglyToad.PdfPig.Content;

public static void Main()
{
    using (PdfDocument document = PdfDocument.Open(@"C:\my-file.pdf"))
    {
        // Returns the total number of pages in the document.
        int pageCount = document.NumberOfPages;

        // Each page is loaded separately by this method.
        Page page = document.GetPage(1);

        // Sizes are given in typographic points (1/72nd of an inch).
        decimal widthInPoints = page.Width;
        decimal heightInPoints = page.Height;

        // The full text of the page as it is defined in the PDF content.
        string text = page.Text;

        // The text and location of each individual letter on the page.
        IReadOnlyList<Letter> letters = page.Letters;

        page = document.GetPage(2);

        // etc..
    }
}

As far as possible the API is restricted to make discoverability easier and immutable to provide proper encapsulation.

Since PDF defines its content for display rather than readability purposes, the page.Text may lack spaces or present words in the wrong order.

For this reason it is necessary for the client to write their own code to form words from the letters provided by page.Letters.

Please bear in mind this is the very first alpha release so highly likely to be unstable.

Installation

The package is available from NuGet. Since it is pre-release in order to see it you will need to tick the pre-release checkbox when searching.

From the Package Manager command line you can install it using:

Install-Package PdfPig -Version 0.0.1-alpha-001

Configuring SonarQube with GitLab and TeamCity

Eliot (Eliot Jones) — Tue, 05 Dec 2017 00:00:00 GMT

Introducing static analysis to a project can help inform code reviews and highlight areas of the code likely to cause errors as well as expose trends in code quality over time. The tradeoff is that there are often many false positives in a report which need to be investigated.

When I configured SonarQube (6.4) to provide static analysis for our C# project we struggled to incorporate it into our normal development process since it sat outside the usual branch -> build -> merge request workflow.

For our source control we were using GitLab (10.1.4) and our build server was running TeamCity (2017.1).

Get the plugin

Gabriel Allaigre has written the sonar-gitlab plugin which enables SonarQube to push its analysis results to GitLab. This presents the results of analysis in the same place we review our merge requests as well as causing build errors when violations occur; and therefore helps incorporate SonarQube into the development workflow.

First you will need to install the sonar-gitlab plugin to your SonarQube environment and follow the steps detailed in the configuration section of the readme:

Set the GitLab URL from the Administration -> Configuration -> General Settings -> GitLab
Set the GitLab user token in the same place. This should be a token for a GitLab user with the developer role. You can get this token in GitLab by going to Profile -> Edit Profile -> Access Tokens and generating a new access token.

Once this is installed the SonarQube configuration is complete.

Configure TeamCity

The installation guide for the sonar-gitlab plugin describes how to configure it when using the GitLab CI or Maven for builds. To run the analysis from TeamCity we need to get some additional information for the parameters to the command line.

If we were running from GitLab's CI we would use the following command to start the Sonar MSBuild Scanner, pushing to GitLab after the analysis completed:

SonarQube.Scanner.MSBuild begin /k:"$env:ProjName" /n:"$env:ProjName" /v:"1.0" /d:sonar.gitlab.project_id=$CI_PROJECT_ID /d:sonar.gitlab.commit_sha=$CI_BUILD_REF /d:sonar.gitlab.ref_name=$CI_BUILD_REF_NAME /d:sonar.analysis.mode="preview"

Since TeamCity does not have access to the GitLab parameters in this command we need to find them in a different way:

$CI_PROJECT_ID

This is an integer ID which uniquely identifies the project in GitLab.

To find out its value you can either go to Project -> Settings -> General settings in GitLab or we can query the GitLab API ourselves. You can navigate to http://$YOUR-GITLAB-URL/api/v4/projects to display the full list of projects. To view a project by ID and ensure you have the correct project you can view by id at http://$YOUR-GITLAB-URL/api/v4/projects/$ID.

$CI_BUILD_REF

The full SHA of the currently building commit. In PowerShell on TeamCity this can be accessed as follows $commit = "%build.vcs.number%".

$CI_BUILD_REF_NAME

The branch name. From PowerShell on TeamCity this can be accessed by: $branch = "%teamcity.build.branch%". To remove the full qualification you can use $branch = $branch.Replace("refs/heads/", "").

Now to start Sonar analysis on TeamCity in a PowerShell step assuming the Sonar MSBuild scanner is in C:\sonar\bin and your project ID is 3 you can run:

$commit = "%build.vcs.number%"
$branch = "%teamcity.build.branch%"
$branch = $branch.Replace("refs/heads/", "")

C:\sonar\bin\SonarQube.Scanner.MSBuild.exe begin /k:"KEY" /n:"PROJECT NAME" /v:"0.0.1" /d:sonar.gitlab.project_id=3 /d:sonar.gitlab.commit_sha=$commit /d:sonar.gitlab.ref_name=$branch /d:sonar.analysis.mode="preview" /d:sonar.scm.provider=git```

Replace the project key, name, version and id values with your values.

It's important to set the analysis mode to preview otherwise the sonar-gitlab plugin will not work.

With the analysis started run your build as normal.

For completing the analysis no GitLab specific changes are required:

C:\sonar\bin\SonarQube.Scanner.MSBuild.exe end

This will then perform the Sonar Scanner analysis, send the results to the SonarQube server which will then forward them to TeamCity.

Hopefully this helps you get TeamCity talking to SonarQube talking to GitLab. Let me know what static analysis tools you use for your C# projects in the comments.

Visual Studio 2017 Red Underline/Incorrect Highlights

Eliot (Eliot Jones) — Thu, 30 Nov 2017 00:00:00 GMT

There are a lot of answers on this topic but in order to aggregate the steps I usually follow for future reference I'm noting them in this blog post.

There are few things more annoying than Intellisense going wobbly and flagging successfully compiling code with errors. Obviously the first step is to restart Visual Studio but if the problem persists you need to try something more. These steps are for Visual Studio 2017 Community with Resharper.

You can check whether the underlines have disappeared after each step or run them all:

Unload then reload the problematic project from Solution Explorer. To do this, right click the project, select Unload Project and then Reload Project. This sometimes helps clear incorrect highlighting due to Resharper especially after merges.
Clear the Resharper cache. This is accessed by going to Resharper > Options > General > Clear caches in the menu. You will need to restart Visual Studio to see if this step worked.
Disable Resharper from Tools > Options > Resharper > Suspend Now. Then start it again from the same location.
Close Visual Studio and then delete the .vs folder from the source folder. This is a hidden folder at the same level as the .sln file.
With Visual Studio closed delete the obj folders from the problematic project folders.

The Curious Case of the Null StringBuilder

Eliot (Eliot Jones) — Mon, 21 Aug 2017 00:00:00 GMT

Today was spent tracking down a very weird bug. In production we were seeing an important part of our document reading fail. We kept getting NullReferenceExceptions when calling AppendLine on a non-null StringBuilder. It didn't prevent us reading the document however the result would be significantly different to the same document on a local instance of our program.

It only started occurring after the production server had been running for a few days which meant we couldn't debug it locally. We were running .NET 4.5.2.

Luckily we had lots of logging to track down the issue. The problem was in a class like this:

public class AlgorithmLogicLogger
{
    private readonly StringBuilder stringBuilder;

    public AlgorithmLogicLogger()
    {
        stringBuilder = new StringBuilder();
    }

    public void Append(string s)
    {
        var message = BuildStringDetails(s);

        stringBuilder.AppendLine(message);
    }

    private static string BuildStringDetails(string s)
    {
        return $"{DateTime.UtcNow}: {s}";
    }
}

This was a class which was originally intended to provide detailed logging for a complicated algorithm.

The call to StringBuilder.AppendLine() inside Append was throwing a NullReferenceException.

After ensuring no weird reflection was taking place and using Ildasm to inspect the compiled code we were sure it wasn't possible for stringBuilder to be null. It was always instantiated in the constructor and never changed elsewhere.

The next working theory was that a multi-threading issue was somehow calling Append prior to the field being set. This was also discounted both because it wouldn't have been possible and also because the code in question was not called from multiple threads.

After ensuring that it wasn't the case that the garbage collector wasn't somehow incorrectly collecting the string builder (because it was only written to, never read, the reading hadn't been implemented yet) we were beginning to run out of ideas.

After combing the production logs from the past 3 days we finally located the issue.

The logger was being created as a singleton:

public class LoggerSingleton
{
    public AlgorithmLogicLogger Logger { get; } = new AlgorithmLogicLogger();
}

Since the StringBuilder was never cleared it would eventually grow larger than the MaxCapacity. When this happened an OutOfMemoryException was thrown and caught further up the stack. Somehow this resulted in a corrupted state which meant further calls to the singleton instance of the StringBuilder such as Append and AppendLine resulted in a NullReferenceException being thrown internally.

Fixing this was fairly simple, just removing the singleton since it was no longer required, but it was odd to see the non-null object throw a NullReferenceException rather than further OutOfMemoryExceptions which is what we observed when the code was replicated locally.

ASP.NET Core Identity Using PostgreSQL

Eliot (Eliot Jones) — Sun, 25 Jun 2017 00:00:00 GMT

Following on from my much older posts about using ASP.NET Identity 2 to manage user accounts in MVC 4 sites, today I needed to use Identity on an ASP.NET Core MVC site.

As with previous versions, the current Identity library for .NET Core 1.1 uses Entity Framework out-the-box. Luckily it's much easier to change this behaviour for the simplest register/log-in flow.

Install

Firstly you need to get the right NuGet package. For the Identity library without Entity Framework this is:

Microsoft.AspNetCore.Identity

Once this is installed in your project you need to provide your own implementation of the IUserStore<TUser> and IRoleStore<TRole>.

Unlike previous versions the TUser no longer needs to implement a specific interface.

For my project I was using PostgreSQL as the database and Dapper.Contrib as the ORM. My User class was simply:

[Table("\"user\"")]
public class User
{
    [ExplicitKey]
    public Guid Id { get; set; }

    public string UserName { get; set; }

    public string NormalizedUserName { get; set; }

    public string Email { get; set; }

    public string PasswordHash { get; set; }
}

(I had to do some faffing around to make sure the table name was detected correctly in my schema but could have called the table anything, it doesn't have to be called "user").

You also need a class for Roles, since I won't be implementing Roles in this tutorial I just created a blank class:

public class Role { }

Fun In Store

Now we need to implement a couple of interfaces.

For users I wanted to support password based log-in.

By default you only need to provide the IUserStore<TUser> implementation, however for almost all applications the behaviours supported by the IUserPasswordStore<TUser> are desired. The IUserPasswordStore<TUser> interface implements IUserStore<TUser>:

public class UserPasswordStore : IUserPasswordStore<User>
{
    private readonly IDataContext context;

    public UserPasswordStore(IDataContext context)
    {
        this.context = context;
    }

    public async Task<IdentityResult> CreateAsync(User user, CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        await context.CreateAsync(user);

        return IdentityResult.Success;
    }

    public async Task<IdentityResult> DeleteAsync(User user, CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        await context.DeleteAsync(user);

        return IdentityResult.Success;
    }

    public void Dispose()
    {
    }

    public async Task<User> FindByIdAsync(string userId, CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        Guid id;
        if (!Guid.TryParse(userId, out id))
        {
            throw new ArgumentException("Id was not a valid Guid: " + userId, nameof(userId));
        }

        return await context.GetByIdAsync<User>(id);
    }

    public async Task<User> FindByNameAsync(string normalizedUserName, CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        var users = await context.GetAllAsync<User>();

        return users.FirstOrDefault(x => string.Equals(normalizedUserName, x.NormalizedUserName, StringComparison.OrdinalIgnoreCase));
    }

    public Task<string> GetNormalizedUserNameAsync(User user, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        return Task.FromResult(user.NormalizedUserName);
    }

    public Task<string> GetPasswordHashAsync(User user, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        return Task.FromResult(user.PasswordHash);
    }

    public Task<string> GetUserIdAsync(User user, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        return Task.FromResult(user.Id.ToString());
    }

    public Task<string> GetUserNameAsync(User user, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        return Task.FromResult(user.UserName);
    }

    public Task<bool> HasPasswordAsync(User user, CancellationToken cancellationToken)
    {
        return Task.FromResult(true);
    }

    public Task SetNormalizedUserNameAsync(User user, string normalizedName, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        user.NormalizedUserName = normalizedName;

        return Task.CompletedTask;
    }

    public Task SetPasswordHashAsync(User user, string passwordHash, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        user.PasswordHash = passwordHash;

        return Task.CompletedTask;
    }

    public Task SetUserNameAsync(User user, string userName, CancellationToken cancellationToken)
    {
        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        user.UserName = userName;

        return Task.CompletedTask;
    }

    public async Task<IdentityResult> UpdateAsync(User user, CancellationToken cancellationToken)
    {
        cancellationToken.ThrowIfCancellationRequested();

        if (user == null)
        {
            throw new ArgumentNullException(nameof(user));
        }

        await context.UpdateAsync(user);

        return IdentityResult.Success;
    }
}

A lot of this code is boilerplate to check for cancellation tokens and to verify arguments, the underlying logic for data access is very basic.

I have not yet implemented the lock-out code in the sample above but it will be fairly simple and similar to the existing code. Just implement IUserLockoutStore<T> in addition to IUserPasswordStore<T>.

The IDataContext class used here is just my custom injectable wrapper around the Dapper/PostgreSQL connection, if you wanted you could directly connect to the database or use whichever data access library you want.

My data context class uses Dapper.Contrib to perform simple CRUD operations:

public class DapperDataContext : IDataContext, IDisposable
{
    private readonly IDbConnection connection;

    public DapperDataContext(IOptions<DatabaseOptions> databaseOptions)
    {
        connection = new NpgsqlConnection(databaseOptions.Value.ConnectionString);
    }

    public async Task CreateAsync<T>(T item) where T : class
    {
        if (item == null)
        {
            throw new ArgumentNullException(nameof(item));
        }

        await connection.InsertAsync(item);
    }

    public async Task DeleteAsync<T>(T item) where T : class
    {
        if (item == null)
        {
            throw new ArgumentNullException(nameof(item));
        }

        await connection.DeleteAsync(item);
    }

    public async Task<T> GetByIdAsync<T>(Guid id) where T : class
    {
        return await connection.GetAsync<T>(id);
    }

    public void Dispose()
    {
        connection.Dispose();
    }

    public async Task<IReadOnlyCollection<T>> GetAllAsync<T>() where T : class
    {
        var items = await connection.GetAllAsync<T>();

        var result = new List<T>();
        result.AddRange(items);

        return result;
    }

    public async Task UpdateAsync<T>(T item) where T : class
    {
        await connection.UpdateAsync(item);
    }
}

You also need to provide an implementation of the IRoleStore<TRole> which for this demo I am ignoring and leaving the methods not implemented. These methods will not be called for our default code, except for the dispose method. For this reason remember to remove the NotImplementedException from the Dispose method:

public class RoleStore : IRoleStore<Role>
{
    public Task<IdentityResult> CreateAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<IdentityResult> DeleteAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public void Dispose()
    {
    }

    public Task<Role> FindByIdAsync(string roleId, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<Role> FindByNameAsync(string normalizedRoleName, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<string> GetNormalizedRoleNameAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<string> GetRoleIdAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<string> GetRoleNameAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task SetNormalizedRoleNameAsync(Role role, string normalizedName, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task SetRoleNameAsync(Role role, string roleName, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }

    public Task<IdentityResult> UpdateAsync(Role role, CancellationToken cancellationToken)
    {
        throw new NotImplementedException();
    }
}

Register Services

Now all that's left to do is register the services at the start up of your MVC or Web API application.

In the ConfigureServices method of the Startup.cs class register your user, role, user store and role store classes as well as any injectable services required by your code.

To register the identity classes correctly, use the AddIdentity<TUser, TRole> extension method. Then register both the user and role store. These method calls must be chained off/follow the call to AddIdentity:

// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
    services.AddIdentity<User, Role>()
        .AddRoleStore<RoleStore>()
        .AddUserStore<UserPasswordStore>()
        .AddDefaultTokenProviders();

    services.AddMvc();

    // Add application services.
    services.AddTransient<IDataContext, DapperDataContext>();

    services.Configure<DatabaseOptions>(Configuration.GetSection("Database"));
}

Calling The Store

Once the services are registered you can request an instance UserManager<User> and/or SignInManager<User> in your controller's constructor. The Identity and Dependency Injection code will handle resolving and registering these based off the store implementations we provided. These then support registration and log-in methods. The log-in method is shown in the controller below:

[Authorize]
public class AccountController : Controller
{
    private readonly UserManager<User> _userManager;
    private readonly SignInManager<User> _signInManager;

    public AccountController(UserManager<User> userManager, SignInManager<User> signInManager,)
    {
        _userManager = userManager;
        _signInManager = signInManager;
    }

    [HttpPost]
    [AllowAnonymous]
    [ValidateAntiForgeryToken]
    public async Task<IActionResult> Login(LoginViewModel model, string returnUrl = null)
    {
        ViewData["ReturnUrl"] = returnUrl;
        if (ModelState.IsValid)
        {
            // This doesn't count login failures towards account lockout
            // To enable password failures to trigger account lockout, set lockoutOnFailure: true
            var result = await _signInManager.PasswordSignInAsync(model.Email, model.Password, model.RememberMe, lockoutOnFailure: false);
            if (result.Succeeded)
            {
                return RedirectToLocal(returnUrl);
            }
            if (result.RequiresTwoFactor)
            {
                return RedirectToAction(nameof(SendCode), new { ReturnUrl = returnUrl, RememberMe = model.RememberMe });
            }
            if (result.IsLockedOut)
            {
                return View("Lockout");
            }
            else
            {
                ModelState.AddModelError(string.Empty, "Invalid login attempt.");
                return View(model);
            }
        }

        // If we got this far, something failed, redisplay form
        return View(model);
    }
}

To easily get controller templates with the usual set of registration and log-in methods supported, simply create a new MVC project from the Visual Studio 2017 template with the Individual Accounts option enabled. Then remove the Entity Framework based Identity library and related classes (such as ApplicationUser) and instead implement the code from this guide.

This gets you the simplest working log-in and registration system, but there's a lot left to implement including the lockout store and the role store as well as 3rd party authentication providers depending on the requirements of your application.

I've put a simple version of this code which uses text files to store data on GitHub.

White Automation

Eliot (Eliot Jones) — Wed, 10 May 2017 00:00:00 GMT

I've spent a few days getting very in-depth with White for WPF UI automation so in order to store the information while it's still fresh in my mind I am writing a blog post.

White is a framework for automating interactions with desktop Windows applications. It is built on top of the UIAutomation library which is part of the .NET framework. The project is currently not very active on GitHub but despite this it's still very good at dealing with desktop interactions on Windows.

Automate all the things

For this tutorial I've decided to automate interactions with the Visual Studio 2015 application; since it's software that most people interested in following this tutorial will probably have.

I'm creating my application as a .NET 4.6.1 console application. Generally White is used for automated UI tests, so you might be using a test project. Whichever project type you choose the information in this tutorial will still be relevant.

First we need to add the TestStack.White NuGet package. Right click your project and select Manage NuGet Packages:

Search the online packages for White and install the TestStack.White package.

Alternatively you can run the installation command from the package manager console.

Install-Package TestStack.White

Once the package is intalled we can start writing code. I have a blank console main method where I'll be putting my code:

public static void Main()
{
    // Writing code here
}

Launch the application

The first thing we need to do is launch the application under test (AUT). The White Application class provides a wrapper over the underlying Windows process.

Add a using statement for the White classes and the process start info:

using System.Diagnostics;
using TestStack.White;

Now we can launch an application, to do this we'll use ProcessStartInfo since it provides us more control over how the application is started, most importantly the Working Directory, otherwise file system access might not work correctly in your application.

The Visual Studio executable is located at the following path for my install:

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe

To launch it we use the following code:

var processStartInfo = new ProcessStartInfo(@"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\devenv.exe")
{
    WorkingDirectory = @"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE"
};

var application = Application.Launch(processStartInfo);

This will run the .exe from its normal location. Once the application is running we can get the windows it opens:

var windows = application.GetWindows();

This method blocks until the application opens a window. For Visual Studio this works well since the splash screen is not counted as a window for this purpose; however for other applications the splash screen may display in its own window.

You could wait for a window with some desired property to load using the following code:

var window = windows.FirstOrDefault(x => x.Name.EndsWith("Visual Studio", true, CultureInfo.CurrentCulture));
while (window == null)
{
    Thread.Sleep(1000);

    window = application.GetWindows().FirstOrDefault(x => x.Name.EndsWith("Visual Studio", true, CultureInfo.CurrentCulture));
}

There are probably nicer ways to do this, but in essence we poll every second to check if any of the application windows have the name we want.

If you need to maximise the window once launched you can use PInvoke. To get the main window handle use application.Process.MainWindowHandle.

Once we have the window we want, it's useful to encapsulate it in another class and centralise all logic that is specific to your UI. This is similar to the Page Object Model pattern for Selenium. The benefit is when the UI changes you have 1 place to change your code. Since the default WPF window is usually called MainWindow I create a MainWindow class to mirror this:

using TestStack.White.UIItems.WindowItems;

public class MainWindow
{
    private readonly Window window;

    public MainWindow(Window window)
    {
        this.window = window;
    }
}

Note that we added another using statement for the Window class.

Now we will use our automated Visual Studio 2015 UI to create a new project. To do this we first need to click File > New > Project. This is shown in the screenshot below:

The set menu

To click these menu items we find each of them by their text and click them in turn:

public void ClickNewProject()
{
    IUIItem file = window.Get(SearchCriteria.ByText("File"));
    file.Click();

    IUIItem newItem = window.Get(SearchCriteria.ByText("New"));

    newItem.Click();

    IUIItem projectItem = window.Get(SearchCriteria.ByText("Project..."));
    projectItem.Click();
}

This requires the followingt using statements:

using TestStack.White.UIItems;
using TestStack.White.UIItems.Finders;
using TestStack.White.UIItems.WindowItems;

Once the Project... menu item is clicked a new modal window opens for the user to choose the project type. This is what we need to interact with next.

New project dialog

We will create a new class to contain interaction logic for the "New Project" dialog window. Imaginatively I will call this NewProjectDialog.

This class will take a reference to its White window in the constructor. This window is different to the main window, but both are of the White type Window.

using TestStack.White.UIItems.WindowItems;

public class NewProjectDialog
{
    private readonly Window window;

    public NewProjectDialog(Window window)
    {
        this.window = window;
    }
}

When we call MainWindow.ClickNewProject() it will return us an instance of the NewProjectDialog. To do this we make use of the built in method White has to get modal dialogs (windows which lock focus and do not let you continue using the main window).

The code for ClickNewProject is modified so it returns an object of the type NewProjectDialog:

public NewProjectDialog ClickNewProject()
{
    IUIItem file = window.Get(SearchCriteria.ByText("File"));
    file.Click();

    IUIItem newItem = window.Get(SearchCriteria.ByText("New"));

    newItem.Click();

    IUIItem projectItem = window.Get(SearchCriteria.ByText("Project..."));
    projectItem.Click();

    // Just take the first since it will (should!) be the only one
    var dialog = new NewProjectDialog(window.ModalWindows()[0]);

    return dialog;
}

Finding yourself

This is where the help of a tool to find elements which understands the UIAutomation visual tree becomes invaluable. The latest guidance from Microsoft seems to be to use their Inspect.exe tool which is included in the Windows 10 SDK. Alternatives include:

UISpy (obsolete)
Snoop
Visual Studio Live Visual Tree for debuggable applications

Since I can't be bothered to spend my evenings working out which combination of options needs to be set just-so for the Inspect tool to install, I instead use Snoop to inspect elements.

For this tutorial however I'll be using the debugger to find elements like some sort of caveman.

Peter Piper picked a project

Our NewProjectDialog will initially support 3 methods.

Select project type. e.g. Class Library, Console Application etc.
Set project name. Set the full name of the project using the textbox.
Click ok. Click the ok button to create the project with the selected settings.

I have scaffolded these out and put the first line in the selecting project type method.

using TestStack.White.UIItems.Finders;
using TestStack.White.UIItems.WindowItems;

public class NewProjectDialog
{
    private readonly Window window;

    public NewProjectDialog(Window window)
    {
        this.window = window;
    }

    public void SelectType(string type)
    {
        var allChildren = window.GetMultiple(SearchCriteria.All);
    }

    public void SetName(string projectName)
    {
        throw new System.NotImplementedException();
    }

    public void ClickOk()
    {
        throw new System.NotImplementedException();
    }
}

In order to find what the item we want to click when we change the project type is I've added this line to the SelectType method:

var allChildren = window.GetMultiple(SearchCriteria.All);

This finds all controls that are a child of the dialog, so is quite slow. When we debug it we find the project types are displayed using labels. Therefore we can change the slow line to instead find the label with the text we want.

var item = window.Get<TestStack.White.UIItems.Label>(SearchCriteria.ByText(type));

This returns the label. We can then click it to select the project type.

item.Click();

Our project type is now selected, rejoice!

Change of name

In order to name the project we need to find the text box used to name the project. Once again we get all items in the window so we know how to refine our search:

var allChildren = window.GetMultiple(SearchCriteria.All);

This time in the debugger we find we need to get the text box with the id "txt_Name". We find the item and set the text:

public void SetName(string projectName)
{
    var textBox = window.Get<TestStack.White.UIItems.TextBox>(SearchCriteria.ByAutomationId("txt_Name"));

    textBox.Text = projectName;
}

Finally to click the "OK" button to create the project the ClickOk method looks like this:

public void ClickOk()
{
    var button = window.Get<TestStack.White.UIItems.Button>(SearchCriteria.ByText("OK"));

    button.Click();
}

Under the hood

Initially when I looked at the File menu I couldn't find a way to locate the menu bar itself. This is because the File menu items are MenuProxy objects instead of the normal menu items.

It's useful to know how to narrow down your element search when the items cannot be found by text or automation id. Generally I use a search for all items:

window.GetMultiple(SearchCriteria.All);

Or find all items of a custom type if I know it's a user control or some 3rd party control:

window.GetMultiple(SearchCriteria.ByControlType(ControlType.Custom));

When you run the debugger to evaluate these statements you can inspect the result. This is the best way to make sure you're only looking at what White can see.

While I was looking for the menu items White didn't seem to be able to find them (before I realised it could if I simply searched by text). For this reason I ended up reinventing the SearchCriteria.ByText searching provided by White. To replicate this functionality I wrote a custom searcher which used the underlying UIAutomation framework. I include it here because it's handy to know how to walk the full tree of UIAutomation AutomationElements.

In order to use the UIAutomation framework code I had to add the following references to my project.

UIAutomationTypes
UIAutomationClient
WindowsBase

To add these right click the project and select "Add Reference...".

Each IUIItem in White such as Window, TextBox, Button, etc, exposes an AutomationElement as a property. This is the class used by the UIAutomation framework.

The code below will recursively walk the controls in the window and find any that contain the text passed in. If it doesn't find a match it will return null. I added the method in MainWindow.cs but it could go anywhere:

private static AutomationElement FindAutomationElement(AutomationElement element, string text)
{
    if (element == null)
    {
        return null;
    }

    // Use the tree walker and start at the root element.
    AutomationElement elementNode = TreeWalker.ControlViewWalker.GetFirstChild(element);

    while (elementNode != null)
    {
        // Check the name property to see if it matches the search item.
        if (elementNode.Current.Name.Equals(text, StringComparison.InvariantCultureIgnoreCase))
        {
            return elementNode;
        }

        // Iterate to next element.
        var value = FindAutomationElement(elementNode, text);

        // If we find something we can exit our recursive method early.
        if (value != null)
        {
            return value;
        }

        elementNode = TreeWalker.ControlViewWalker.GetNextSibling(elementNode);
    }

    // No more items in this branch of the tree.
    return null;
}

The using statements required in MainWindow.cs for this code are:

using System;
using System.Windows.Automation;
using TestStack.White.UIItems;
using TestStack.White.UIItems.WindowItems;

Finally I used this method to right click the references node in solution explorer and add a reference to my new project. This method which I put in MainWindow.cs shows how to use White to navigate the UI and filter searches by searching from child objects.

public void AddReference(string referenceName)
{
    var solutionExplorerTab = window.Get(SearchCriteria.ByText("Solution Explorer"));

    // Focus the tab
    solutionExplorerTab.Click();

    // Get the solution explorer tree control
    var solutionExplorerTree = window.Get<Tree>(SearchCriteria.ByAutomationId("SolutionExplorer"));

    // The References node seems to throw an error when clicked normally, so I use my custom method.            
    var node = FindAutomationElement(solutionExplorerTree.AutomationElement, "References");

    var nodeItem = new UIItem(node, window);

    // Focus
    nodeItem.Click();

    // Click
    nodeItem.RightClick();

    // The context menu is accessed as a popup on the window.
    window.Popup.Item("Add Reference...").Click();

    // Get the only modal window.
    var dialog = window.ModalWindows()[0];

    // Click the list item for the reference type to activate it and reveal the checkbox.
    var listItem = dialog.Get(SearchCriteria.ByText(referenceName));

    listItem.Click();

    var checkBox = listItem.Get<CheckBox>(SearchCriteria.ByControlType(ControlType.CheckBox));

    checkBox.Checked = true;

    // Click "OK" to close the dialog.
    dialog.Get<Button>(SearchCriteria.ByText("OK")).Click();
}

The full code

You can find the final code for this tutorial on GitHub.

How it works: Selenium

Eliot (Eliot Jones) — Thu, 04 May 2017 00:00:00 GMT

I've done quite a bit of work with Selenium on and off, mainly for browser automation tests. There are many high quality official libraries for Selenium covering a range of languages. It's an amazing technology and a real joy to work with; however sometimes stuff can go wrong and it's useful to know what's going on under the hood in order to debug where the problem lies.

For this reason I present my very-probably-wrong guide to Selenium.

Gecko, drivers!? What's going on?

There are 3 components in code that interacts with Selenium (ignoring RemoteWebDriver use cases):

Your code
A driver
The browser

For each browser that supports Selenium, the browser vendor provides a "driver". This is usually a small executable (.exe) program tailored for a specific browser:

Chrome / Chromium
Firefox
Edge
Other drivers available

This driver is the part your code actually interacts with. The driver then communicates with the browser using whatever magic the vendors design. In order to change browsers while using the same code, the drivers expose interfaces/endpoints that implement the WebDriver specification.

Put simply, this means the drivers are a simple RESTful API.

Demo

To demonstrate this I'll use the ChromeDriver version 2.29 which is the latest version at the time of writing. Drivers can only talk to the versions of the browser they are compatible with. In this demo I'm running Chrome on version 58.

Once you have downloaded the driver you get a simple program called chromedriver.exe. If you double click the exe it launches a console window with some information about the driver version you are running and what port it's listening on:

On start-up the driver logs the message:

Starting ChromeDriver 2.29.461591 (62ebf098771772160f391d75e589dc567915b233) on port 9515

This tells us we can access the driver at the URL http://localhost:9515.

Now that the driver is listening we can ask it to launch a browser instance for us. To do this we need to send an HTTP POST request to the following URL:

http://localhost:9515/session

This method is defined in the specification. We also need to provide some extra information about the capabilities we want from the browser instance. To do this we pass JSON in the request body.

For a blank set of capabilities the JSON looks like this:

{
    "desiredCapabilities": {}
}

To interact with the driver using HTTP requests I am going to use PowerShell since it's installed on all Windows systems. You can use Postman or Fiddler or curl or whichever client you want.

After launching PowerShell we define the JSON for the request body and then send the request using PowerShell's Invoke-RestMethod:

$body = @{
    desiredCapabilities = @{}
}

# Change the body object to something we can use to send a request
$json = $body | ConvertTo-Json

# Send the actual request
$response = Invoke-RestMethod -Method Post -Uri http://localhost:9515/session -Body $json

After sending the last line, a new browser window should open. If you don't see the window try typing $response.value into the PowerShell window and hitting enter. This should contain a message telling you what went wrong.

This window has a unique SessionId we use to interact with it specifically. We can display this id and assign it to a variable using PowerShell:

# Store the id
$sessionId = $response.sessionId

# Show us what it is
Write-Host $sessionId

For Chrome the session Id is some hash, for example 3c0802805fbec93515dacdcc2e6bba72. For Edge it's a GUID.

Now we have a blank window, the next step is to navigate to a web page. From the spec we need to hit the URL http://localhost:9515/session/{session id}/url where {session id} is the actual id of the session we created (and helpfully stored in the $sessionId variable). We also need to tell the driver what URL we want to navigate to. Again we provide this information in the request body as JSON:

{ "url" : "https://w3c.github.io/webdriver/webdriver-spec.html#go" }

To achieve this in PowerShell we do the following:

$body = @{ url = "https://w3c.github.io/webdriver/webdriver-spec.html#go" }

$json = $body | ConvertTo-Json

# PowerShell will replace $sessionId in the URL with the actual session Id
$response = $response = Invoke-RestMethod -Method Post -Uri http://localhost:9515/session/$sessionId/url -Body $json

After the last line the browser should navigate to the page in the specification. Finally let's find and click a button. Based on the information from the specification:

# Define how we want to search for the element, in this case the element with the id "respec-pill"
$json = @{
    "using" = "id"
    "value" = "respec-pill"
} | ConvertTo-Json

# Find the element using the search parameters defined above
$response = Invoke-RestMethod -Method Post -Uri http://localhost:9515/session/$sessionId/element -Body $json

# Store the ID of the located element
$elementId = $response.value.ELEMENT

# Click the element using the ID we found, no need to post any JSON for this call
$response = Invoke-RestMethod -Method Post -Uri http://localhost:9515/session/$sessionId/element/$elementId/click

Hopefully this shows you that the Selenium interaction that occurs when your code uses a library for Selenium is actually (fairly) simple. The complicated and magic part happens when the driver talks to the browser.

Some key parts which can go wrong are:

Your code is using a version of the Selenium library which is sending requests that the driver doesn't understand.
Your driver is the wrong version to interact with the version of the browser.
The driver you are using has a bug or does not yet support a feature of the WebDriver specification (for example Edge supports a quite limited subset at the moment).
Your code has a bug.

Of these, the last item is the absolute, 99.9%, most likely culprit for any bug. Followed by a version mismatch between any 2 of the components. Using your knowledge of how to interact with the driver without any intermediate code you can now perform an extra check to verify where the problem likely lies.

Writing a custom debug visualizer for Visual Studio 2015

Eliot (Eliot Jones) — Sun, 20 Nov 2016 00:00:00 GMT

Often at work I'll be debugging some incredibly complex logic with lists that can be hundreds, if not thousands, of items long. While the ability to inspect these lists using the normal debugging tools is useful, it gets annoying trying to scroll to the 700th item in a list and then accidentally moving the mouse focus away and having to start again.

For this reason I wanted to look into the possiblity of creating a custom display for certain objects while debugging in Visual Studio. For those of you who have worked with the DataTable you might be familiar with this screen:

This can be accessed by hovering over the object when paused in the debugger and clicking the magnifying glass icon as shown here:

Visual Studio supports creating custom screens like this called Visualizers.

Getting Started

The linked MSDN guide and similar guides detail how to create the custom screen using WinForms. I wanted to use WPF because it was the desktop UI technology I was most familiar with. I followed the walkthrough but used this code to display the Visualizer:

var window = new MainWindow();
window.ShowDialog();

This worked well and my empty WPF window was displayed, but when I closed the window a CannotUnloadAppDomainException was thrown and the WPF window reopened:

System.CannotUnloadAppDomainException was unhandled
HResult=-2146234347 Message=Error while unloading appdomain. (Exception from HRESULT: 0x80131015)

I spent a couple of hours trying to figure this out and ended up talking to myself on StackOverflow, in the end it turned out it was due to the Stylus Input thread refusing to close.

The solution to this was to explicitly close the WPF Dispatcher when closing the window:

protected override void OnClosing(CancelEventArgs e)
{
    Dispatcher.InvokeShutdown();

    base.OnClosing(e);
}

This closed the stylus input (touchscreen) thread properly and my Visualizer now worked properly.

Custom Objects

Since I wanted to write Visualizers for custom types rather than just the system types, the next step was to experiment with a custom class. For testing I imaginatively created a person class:

public class Person
{
    public Guid Id { get; private set; }

    public string FirstName { get; private set; }

    public string LastName { get; private set; }

    public string FavoriteColor { get; private set; }

    public DateTime BirthDate { get; set; }

    public Person(string firstName, string lastName, DateTime birthDate, string favoriteColor)
    {
        Id = Guid.NewGuid();
        FirstName = firstName;
        LastName = lastName;
        BirthDate = birthDate;
        FavoriteColor = favoriteColor;
    }

    private Person()
    {
    }
}

Generally if a class is marked with the [Serializable] attribute it can be passed directly to your Visualizer. This is because the active object from Visual Studio must be serialized and passed to the debugger side which runs separately, the Visualizer doesn't have access to the original object. When the type is marked with the serializable attribute it can be easily retrieved using the method:

objectProvider.GetObject();

However since a lot of the classes I was working with weren't going to be [Serializable] I needed to support types without this attribute.

Non-Serializable types

The Person class above is not marked as [Serializable] in order to experiment with visualizing non-serializable types.

Since we need to pass objects between the process being debugged and the Visualizer I opted to use JSON.

My Visualizer code was:

assembly: DebuggerVisualizer(typeof(DebuggerSide), typeof(PersonObjectSource),
    Target = typeof(Person), Description = "Person Visualizer")]
namespace Visualizer
{
    public class DebuggerSide : DialogDebuggerVisualizer
    {
        protected override void Show(IDialogVisualizerService windowService, IVisualizerObjectProvider objectProvider)
        {
            try
            {
                using (var reader = new StreamReader(objectProvider.GetData()))
                {
                    var result = reader.ReadToEnd();

                    var person = JsonConvert.DeserializeObject<Person>(result, new JsonSerializerSettings
                    {
                        ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor,
                        ContractResolver = new PrivatePropertyResolver()
                    });

                    if (person == null)
                    {
                        MessageBox.Show("Cannot visualize this object, please use a different visualizer.", "Type Not Supported", MessageBoxButtons.OK, MessageBoxIcon.Information);

                        return;
                    }

                    var window = new MainWindow(person);

                    window.ShowDialog();
                }
            }
            catch (Exception ex)
            {
                Trace.Fail(ex.Message, ex.StackTrace);
            }

        }

        public static void TestShowVisualizer(object thingToVisualize)
        {
            var visualizerHost = new VisualizerDevelopmentHost(thingToVisualize, typeof(DebuggerSide), typeof(PersonObjectSource));
            visualizerHost.ShowVisualizer();
        }
    }
}

It's worth noting that this handles the case where it's been handled the wrong object type gracefully using a WinForms message box to inform the user the object is wrong:

if (person == null)
{
    MessageBox.Show("Cannot visualize this object, please use a different visualizer.", "Type Not Supported", MessageBoxButtons.OK, MessageBoxIcon.Information);

    return;
}

This turns out to be useful for a List visualizer where, even if only want to provide visualization for a single type of object, you can't specify the type of T in a Visualizer for List<T> so you must handle Visualization requests for the wrong type.

In order to provide the Person to this Visualizer we must write it to the stream. We do this by specifying that our Visualizer uses a PersonObjectSource. This class inherits from the normal VisualizerObjectSource but overrides GetData to use JSON serialization instead:

public class PersonObjectSource : VisualizerObjectSource
{
    public override void GetData(object target, Stream outgoingData)
    {
        var person = target as Person;

        if (person == null)
        {
            return;
        }

        var writer = new StreamWriter(outgoingData);

        var content = JsonConvert.SerializeObject(person);

        writer.WriteLine(content);

        writer.Flush();
    }
}

Again we use the as operator here to handle the case where we get an unexpected or null object type.

Now we have our person object in our Visualizer we can use it in our WPF window and write a normal WPF application. For this example I put together a terrible UI:

Lists

I also wanted to support Visualization of lists of a certain type. Unfortunately it's not possible to specify the type of List<T> you wish to provide Visualization support for so Visual Studio will provide your visualizer for all Lists.

Since the Visualizer I was intending to write would be internal to the company it would be acceptable to handle the case where the List was of the wrong type by displaying a message box, but this would be annoying to general users.

To specify that your Visualizer supports Lists you simply specify this in the assembly attribute:

[assembly: DebuggerVisualizer(typeof(DebuggerSide), typeof(VisualizerObjectSource),
    Target = typeof(List<>), Description = "My List Visualizer")]

And then handle being passed the wrong type in the Show method.

A more advanced case

I am terrible at both pronouncing and writing Regexes (Ree-gexes?) and struggle with cases more complex than .*. My favourite site for writing Regexes is Regex Storm since it is based on the .NET regex engine and provides a clean, simple UI.

Despite having that site available I still struggle with getting Regexes right, especially when using named Groups.

For that reason to practice writing Visualizers I put together this basic Visualizer for a MatchCollection:

The full code for this Visualizer and the simpler Person Visualizer can be found here.

Installation

In order to install a Visualizer the .dll simply needs to be placed in (assuming a default Visual Studio installation):

C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Packages\Debugger\Visualizers

You will need an administrator account to move files into this folder.

Reading from a COM port

Eliot (Eliot Jones) — Mon, 14 Nov 2016 00:00:00 GMT

I was recently researching how to read from a COM (Serial) port in order to communicate with an old embedded system. If you're not familiar with them that's one of these:

C# provides the SerialPort class for just this purpose which is hugely helpful. However there's one problem with this class which is that it's just not... very good. I didn't know about this but this post by Ben Voigt should leave you in no doubt that it has some problems.

Setup

In order to test the code on my machine without access to the actual device which was elsewhere, I installed Eltima's Virtual Serial Port Driver which would allow me to test with emulated serial ports. Once I had set up a pair of ports I could send data from one end using PowerShell.

I created a pair of virtual ports called COM1 and COM2, COM1 would be the port I was using C# to read from. COM2 would emulate the device sending data.

For testing I was just sending blocks of text from COM2 using the following PowerShell script:

$port = New-Object System.IO.Ports.SerialPort COM2,9600,None,8,one
$port.Open()
$port.WriteLine("some test data")

The Code

Because the application I was using to read from the port would be a C# console application I didn't have any way to use async-await properly so stuck with the approach outline in Ben's blog post, namely using IAsyncResult.

The code which listens to and reads from the result looks like this:

public class COMReader
{
    private static readonly BlockingCollection<byte[]> queue = new BlockingCollection<byte[]>();

    public static int Main()
    {
        using (var port = new SerialPort("COM1", 9600, Parity.None, 8, StopBits.One)
        {
            Handshake = Handshake.None,
            ReadTimeout = 1000,
            WriteTimeout = 1000
        })
        {
            port.Open();

            var consumer = new Consumer(queue);
            var buffer = new byte[256];
            Action startListen = null;
            var onResult = new AsyncCallback(result => OnResult(result, startListen, port, buffer));

            Task.Run(() => consumer.Start());

            startListen = () => 
            {
                port.BaseStream.BeginRead(buffer, 0, buffer.Length, onResult, null);
            };

            startListen();

            while (true && port.IsOpen)
            {
                // handle user's console window interaction.
            }

            queue.CompleteAdding();

            if (port.IsOpen)
            {
                port.Close();
            }
        }

        return 0;
    }

    private static void OnResult(IAsyncResult result, Action startListen, SerialPort port, byte[] buffer)
    {
        try
        {
            if (!port.IsOpen)
            {
                return;
            }

            var actualLength = port.BaseStream.EndRead(result);

            var received = new byte[actualLength];

            Buffer.BlockCopy(buffer, 0, received, 0, actualLength);

            queue.Add(received);
        }
        catch (IOException)
        {
            Console.WriteLine("I/O exception encountered. Closing.");
            port.Close();
            queue.CompleteAdding();
            return;
        }

        startListen();
    }
}

Most of this is fairly self explanatory, we create a SerialPort instance (in a using statement since it implements IDisposable) and then open it. We also create a new Consumer, which I will talk about in a moment.

Next as in the blog post I mentioned we have an action to execute, this is declared separately to its assignment since it needs to run itself when it finishes reading. The body of the code to execute once the async read finishes is in the OnResult method. This just reads the current bytes from the port's stream.

The next step is to start the Consumer running on another thread using Task.Run(). Then we simply start the action which listens to the port.

This will listen for incoming messages on the port and each time something is received it will call the code in OnResult(..) and then start listening again.

Consuming

In order to read the output data two things have to be done:

Return to the startListen() method at the end of OnResult as quickly as possible to avoid missing data.
Notify some subscriber to the data and work out if the received data forms a complete message.

Since the SerialPort will read data whenever it feels like it instead of waiting for a complete message we need to know what our definition of a "message" is, in the case of my PowerShell script its end is the \n character.

This is an almost perfect use case for a BlockingCollection, this provides a location for the OnResult method to dump the data and allows us to write a consumer for the data which only runs when new data is found.

Our Consumer class is constructed by passing in the blocking collection queue which the main method writes to:

var consumer = new Consumer(queue);

The code for consumer is below:

public class Consumer
{
    private const byte TerminatingCharacter = 10;

    private readonly BlockingCollection<byte[]> producer;

    private readonly List<byte> bytes = new List<byte>();

    public Consumer(BlockingCollection<byte[]> producer)
    {
        this.producer = producer;
    }

    public void Start()
    {
        Console.WriteLine("Start listening");

        foreach (var item in producer.GetConsumingEnumerable())
        {
            ProcessInput(item);
        }

        Console.WriteLine("Finish listening");
    }

    private void ProcessInput(byte[] input)
    {
        if (input == null || input.Length == 0)
        {
            return;
        }

        if (input[input.Length - 1] == TerminatingCharacter)
        {
            var message = Encoding.UTF8.GetString(bytes.Concat(input).ToArray());

            Console.WriteLine($"Message Received: {Environment.NewLine}{message}{Environment.NewLine}");

            bytes.Clear();
        }
        else
        {
            bytes.AddRange(input);
        }
    }
}

This class keeps a reference to the producer's blocking collection. When it starts listening using the Start method it will keep listening to the collection until queue.CompleteAdding(); is called. Every time a new item is added to the queue the consumer will run the ProcessInput method where item is the byte array we placed on the queue in the producer.

Using this approach we can perform a slower analysis of the messages we have received so far in order to pull the complete messages out. For this demo application I just check for the last byte received to be the character \n but your definition of message may be more complex depending on the device which produces the message.

Because we run the consumer on its own thread:

Task.Run(() => consumer.Start());

The time taken for the listener to respond to new data from the port is completely decoupled from the code to process these events.

This isn't perfect code because in the failure case it doesn't wait for the consumer to finish processing the queue before shutting down and there is insufficient error handling around parts of it. However it will hopefully provide an idea how the BlockingCollection can be used to communicate between a publisher and subscriber on separate threads and how that principle can be applied to Serial Ports.

4096 limit

It's worth noting that by default a message is limited to 4096 bytes, if the end of the message isn't received in this time the rest of the message will be dropped. In order to avoid this set the ReadBufferSize on the SerialPort to something higher than 4096.

I'm not an embedded systems person so all of this is my best guess, if I've done something completely idiotic, please let me know in the comments.

New Site

Eliot (Eliot Jones) — Sat, 05 Nov 2016 00:00:00 GMT

I've not been able to find time to blog at all since starting my new job in March, working at a startup means learning a lot but also working long hours!

Almost a year and a half after first blogging about vNext/.NET core I've finally written my first (almost) functional program in it. I've changed this blog from a horrible mess of PHP to a nice MVC6 site running on an Ubuntu server using .NET core. The nice part is noticeable in the code, the UI still remains a dumpster-fire.

The motivation to finally get a site running using .NET core was from this blog post by Scott Hanselman which shows just how easy it is to get something running on .NET core. I grabbed a very cheap Ubuntu droplet from Digital Ocean and followed along, all development was done on my awful 8 year old laptop running xUbuntu. It's so nice to be able to build C# apps and have them run across platforms.

Another improvement is I have scripted my entire deployment using a shell script, as soon as I'm happy with my changes I can run one script which will build, package, deploy and start the site. No more tedious manual FTP interaction.

The code for the blog is on GitHub so you can see just how badly written it is and probably hack my terrible home-brew security stuff.

I'm still missing a couple of features from the old blog which I was too lazy to port across but hopefully they won't be missed by anyone, if you want them back (code snippets hosted on this site for some older tutorials) let me know in the comments.

Run tests task for Elixir in Visual Studio Code

Eliot (Eliot Jones) — Fri, 12 Feb 2016 00:00:00 GMT

In order to learn Linux I'm biting off more than I can chew by trying to learn Elixir at the same time. From the available code editors for Linux I'm using Visual Studio Code.

Visual Studio Code has the ability to configure custom tasks.

To get to the custom task configuration you can use the command palette with Ctrl + Shift + P and type Tasks: Configure Task Runner. This will open "tasks.json". Replace the contents with:

{
    "version": "0.1.0",
    "command": "bash",
    "isShellCommand": true,
    "showOutput": "always",
    "args": [
        "-c"
    ],
    "tasks": [
        {
            "taskName": "Elixir-Test",
            "suppressTaskName": true,
            "isTestCommand": true,
            "args": ["mix test"]
        }
    ]
}

And save the file. Now if you use Ctrl + Shift + T mix will run your tests for the project (assuming you have the mix.exs level folder open) and print the output to the console.

This command could probably be improved by checking that it's executing in the correct folder first but I'm nowhere near confident enough with Bash to write it yet.

Further notes from adventures in Elixir

Function capturing (Capture operator)

This concept took me a little while to get to grips with because I couldn't think of an equivalent from C#.

A typical anonymous function in Elixir might take this form:

square = fn x -> x * x end

You would then call it with the dot "." between the function name and the argument like this:

# Outputs 9
square.(3)

The function capturing operator (&) allows us to express the same function even more quickly:

square = &(&1 * &1)

We call this function in the same way and the output is exactly the same:

# Outputs 9
square.(3)

You could break this down further to understand it:


# Start anonymous function definition with unknown number of arguments.
# e.g. fn ? ->
&(

# Take the first argument (&1) to the anonymous function and multiply by the same argument
&1 * &1

# End anonymous function
)

If you try to pass more than one argument to this function you will get an error:


# (BadArityError) function with arity 1 called with 2 arguments (0, 3)
square.(0, 3)

The arity is the number of arguments the function takes.

Equally if you try to define the function without using all the positional arguments in the function you get a compile error:

# Won't compile
square = &(&2 * &2)

There is another form of function capture which only works with named functions. For example for the standard anonymous function (using the length named function from the String module):

len = fn s -> String.length(s) end

Running that will give the following output:

# Outputs 9
len.("chow chow")

Using our current understanding of the capture operator we could rewrite to the equivalent but shorter form:

len = &(String.length(&1))

There is a second form we can use which is equivalent:

len2 = &String.length/1

We can also verify these functions are identical:

# Outputs true
len == len2

Git for beginners

Eliot (Eliot Jones) — Mon, 08 Feb 2016 00:00:00 GMT

Out of the different Source Control options Git is definitely the one most worth learning, however, it's also one of the most confusing.

With that in mind I'm going to add to the crowded market of "Git beginners" tutorials. Hopefully given that I can still remember the struggle of learning it and that my current level of understanding probably rates as "simpleton" this will be a useful tutorial.

I'm going to be writing for Git on Windows and will try to assume no existing knowledge beyond knowing what a file and folder is. Because of this I'm going to make a lot of oversimplifications.

Install It

Use the download link here or Google "Windows Git" and click the first link.

After using the installation defaults, you're done.

Where does it install?

Depending on which architecture version (x86/x64) you installed it will probably install in either C:\Program Files\Git or C:\Program Files (x86)\Git.

Launch It

Since using one of the nice user interfaces will only shield you from the true horror, it's best to start using console.

Search for "Git Bash" in your start menu or find it in the install location. Once launched you should see this window:

Depending on how comfortable you are with Window's command prompt you're either happy with what's going on, or the panic is already setting in.

Bash

Bash is effectively the Linux version of the command prompt from Windows. The command prompt is just a window which you type stuff into and things happen.

The dollar sign "$" is the prompt, it's asking you to write something on that line and when you hit enter it will run your command.

Let's start by working out where we are. Type the following command at the prompt:

pwd

And hit enter. You should see something like the following result:

By using pwd (print working directory) we're able to see that our current command prompt (Bash) is pointing at "C:\Users\Eliot" (or your equivalent)

Why does the file path look weird?

This is because Windows decided to be difficult and use a different way of showing folder and file locations to everyone else. Since Bash is like running Linux it uses forward slashes "/" instead of back slashes "\".

Now let's move somewhere useful.

Since I'm exceedingly lazy I like to store all my code directly on my C drive, rather than in my Users folder. I have a folder located at C:\code (for the sake of the tutorial you should create one too).

For navigating folders Git Bash uses the same command as Windows. cd -change directory. Go ahead and type the following command:

cd /c/code

Your console output should look similar to this:

Notice how the yellow tilde character "~" above the prompt has changed to be "/c/code"? From this we learn that the yellow text shows us our currently active folder. To verify this you could type pwd and check it prints out "/c/code".

This also tells us that the "~" character has a special meaning. It corresponds to the base folder of the user's environment, mine is located at "C:\Users\Eliot".

So far we've not actually used any Git commands, these have been Bash commands. Git commands can are run through Bash.

Gitting Started

If you're still here after that terrible pun, it's time to actually do something with Git.

Let's create some code we might want to store under source control. For this demo I'm just going to make a project containing some static HTML and css.

In a new folder located at C:\code\my-website I made an html page and some css (index.html and site.css). Create the folder called my-website now and add these files. You can leave them empty for now if you don't know how to write html or css:

To prepare to use our first Git command, move into this folder using Bash. If your current directory is C:\code you can type:

cd my-website

Or to get to it from anywhere:

cd /c/code/my-website

Now for our first Git command, type:

git init

You should see the following output:

Initialized empty Git repository in C:/code/my-website/.git/

This is how we create a Git repository.

A Git repository is a basically a folder which is under source control.

How does the repository work?

If you look at the C:\code\my-website folder in Windows file explorer you should see a new folder called ".git" alongside your files. If you can't see it you may need to enable viewing hidden folders.

This .git folder contains the files Git uses to track changes under source control. This makes it easy to delete a repository while keeping the files. All you need to do is delete the .git folder.

There's a new bit of information in the line above the prompt in Git Bash. After the current active path there's the word "master" in brackets. This is the current branch name.

Don't worry too much about branches for now, we'll get to them in a bit.

All we have to do to make our repository active in Bash is navigate to the folder containing the repository (or any folders under that folder). If we use the cd command to navigate up a level and then back into our my-website folder, we see the branch name disappears then reappears from the text above the prompt. The branch name only shows when we're in a repository. Try these commands:

cd ..
cd my-website

For an illustration of what you should see, look at the screenshot below:

Fear of Commitment

We've know that the Bash console shows us which branch is currently active when we're in a repository. Another way to get this information is the command:

git status

As well as printing the branch name this command also prints the status of files in the folder (repository):

On branch master

Initial commit

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        index.html
        site.css

nothing added to commit but untracked files present (use "git add" to track)

This tells us we have 2 untracked files. An untracked file is one which isn't under source control. If we delete it, or mess it up in some way, Git isn't going to help us. With that in mind let's track it. As the output from git status helpfully suggests, we can use the following command:

git add

If you type just that statement you get the following output:

Nothing specified, nothing added.
Maybe you wanted to say 'git add .'?

The argument to git add is the path to the file or files you want to track. The '.' dot means "all currently untracked files". Let's use the following two commands to add the files and then check the status:

git add .
git status

Our changes are now "staged" and we're ready to commit them. A commit is a snapshot of how all files in the project looked at a particular point in time. It's a bit like if we copied our project folder and renamed it "my-website-20160209".

Once you have a commit you can always go back to how the files looked when you made that commit.

This is an exciting moment for us, the first commit in our repository! The simplest way to make a commit is with the following command:

git commit -m "our first commit"

The -m option tells Git we will provide the message for our commit along with the command. This message can be anything; "toast", "tire iron", "updated tests to use mocks". It's just a friendly label we can use to remind people and ourselves what we did between this commit and the last one.

Now if we use git status we can see that nothing has changed in our repository since we made our commit.

Captain's Log

Now we've made a commit how do we view the commits in our repository? The answer is:

git log

The log lists all the commits in our repository with the latest at the top of the log. The long yellow word containing both numbers and letters is the hash of our commit. This is a unique bit of text which identifies our commit for Git commands.

From Logs to Branches

I mentioned earlier that the "master" in brackets tells us which branch we're on. Now we've got our first commit it's time to learn about branches.

A branch is a way of having multiple different sets of the same code which are changing in different ways. For example if you have a text file (file.txt) that both you and your friend Eric are editing, you might share a folder containing "file.txt, my-file.txt, eric-file.txt". This way you can work on your file, Eric can work on his and neither of you are going to irreversibly mess up the original file.

Every Git repository starts with a branch called "master". That's why, when you used the git init command, the active branch name was "master".

To see every branch in your repository you can type:

git branch -a

The -a option tells Git to list all branches. The output of this command is:

* master

The asterisk '*' shows which branch is currently active.

Now let's assume we want to change our files, but these are experimental css changes which we might throw away. To create a new branch:

git checkout -b css-changes

The -b option means this command will create a new branch, rather than moving between existing branches.

If you type git branch -a now, you'll see that there are two branches and the branch css-changes is active. The text above the bash prompt will also have updated:

If we use git log to check our commit history we can see it's the same as master. Creating a new branch from another branch will bring all the commits from the source branch.

Now to make some changes to a file update site.css to add some styles to it, for instance:

div{
    box-sizing: border-box;
}

Once you save these changes in your editor, type git status to see what's going on.

On branch css-changes
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   site.css

no changes added to commit (use "git add" and/or "git commit -a")

Git has detected that our site.css has changed. We need to use the git add command again to be able to commit the file. Use the following 2 commands:

git add .
git commit -m "css changes"

When we type git log we now see 2 commits in this branch.

After making this commit let's also change the index.html file. I added the line:

<iframe></iframe>

Now this was a mistake, iframes are bad. How do we undo this? If you type git status the output should say something like:

On branch css-changes
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

        modified:   index.html

no changes added to commit (use "git add" and/or "git commit -a")

One option is to use git checkout -- index.html which will reset the file. However we could also reset everything to be like the last commit.

Reset

In order to reset our branch to be like the last commit we can (unsurprisingly) use git reset. First you need the hash of the commit you want to return to. Use git log and select the first 6-9 characters of the hash of the top commit in the log. For example my output is:

commit dd1f49b24a190c2c28e42a361e7e8ddaa697aab4
Author: Eliot Jones <elioty@hotmail.co.uk>
Date:   Tue Feb 9 19:00:33 2016 +0000

    css changes

commit 14eff26985b984143cadf32f1a3bd111f7f6c7e2
Author: Eliot Jones <elioty@hotmail.co.uk>
Date:   Tue Feb 9 18:38:32 2016 +0000

    our first commit

Your commit hashes will be different, select the first part of the top one. In my case the text I want is "dd1f49b". Next we can use a hard reset to get our branch looking like it was at the time of that commit:

git reset --hard dd1f49b

Note that the hard reset command is dangerous and you could lose work, it's generally better to use git checkout -- <file>.

Now if we type git status we can see that we have no changes in our branch.

Recap

So far we've:

Created our repository
Made our first commit on master branch
Branched from master to css-changes
Made another commit on the css-changes branch
Discarded some unwanted changes on the css-changes branch by rolling back to a previous commit

If we illustrate commits with circles and branches with lines, then our current branches and commits look like this:

The beauty of Git is we can go back to master branch and interact with our code as it looked before we made the commit to change the site css in our css-changes branch. To do this we don't even need to do a hard reset. Use the command:

git checkout master

This change your active branch and set your active view of the repository back to how the code looked in the master branch.

Where are the files in the css-changes branch?

One of the things which differs in Git to other version control systems is that there's only ever a single version of a file on disk at a given time. Git doesn't store branches in separate folders, everything is stored in Git's database. This means that branching is an incredibly low cost operation, you can make many branches with no real memory or performance issues.

Merge

Now we've verified we're happy with the code in our css-changes branch it's time to merge it into master. Generally you want to reserve master for completed "shippable" code. This means you merge into master when a commit (or series of commits) is ready to be used/deployed/released.

Given our current active branch is master all we have to do to complete the merge is enter the command:

git merge css-changes

Now one last time let's type git log to see our commit history:

After the merge the changes from our css-changes branch have been brought into master. Effectively we've copied the commit from css-changes to master. We still have the css-changes branch, you can verify this using git branch -a.

The merge operation git merge <branch> will transfer any commits with different hashes from the branch passed as the argument to the currently active branch.

Wrapping Up

That concludes this post. In the next post we'll take a look at working with remotes, using GitHub as our example and also rebasing, merge-conflicts and more.

Feel free to point out any terrible errors or missing information in the comments.

Fixie Data Driven tests

Eliot (Eliot Jones) — Thu, 04 Feb 2016 00:00:00 GMT

I tend to spend all my coding time nowadays plagued with guilt. I try to be a good developer and refactor the code, after running the tests, after writing the code, after running the tests, after writing the tests. But in the interests of getting any code written at all, I end up skipping between one and all of these steps (generally not the writing code part).

If this sounds like you, I cannot recommend Fixie strongly enough.

It's a low ceremony test framework that presents the minimum possible barrier to entry for the guilt driven tester/developer.

Positives

Fixie is so nice for a few reasons. Firstly it's self contained. After installing from NuGet, the Visual Studio test runner for Fixie is already set up.

This means you don't have to locate the correct package for your Visual Studio version. If you're lazy like me then you'll appreciate this time-saver.

To install from the package manager console use:

Install-Package Fixie

Or search "fixie" on the NuGet package manager.

The next point in its favour is your tests are just code. While MSTest and to some extent NUnit are an alphabetti spaghetti of attributes which challenge you to remember the difference between your [TestFixture] and [TestClass] attributes, Fixie requires no attributes to start working.

By default any public void or public async Task method is detected as a test as long as the parent class name ends with "Tests". Test set-up and tear-down are handled by constructors and IDisposable.

An example test:

public class OrderDateTests : IDisposable
{
    private readonly DateTime frozenDate;

    public OrderDateTests()
    {
        frozenDate = new DateTime(2016, 02, 04);
        SystemTime.Freeze(frozenDate);
    }

    public void OrderDateCannotBeInFuture()
    {
        try
        {
            var order = new OrderDate(frozenDate.AddDays(7));
        }
        catch (OrderDateException)
        {
            // Could run assertion here if desired
        }
    }

    public void OrderDateCanBeCurrentDate()
    {
        var order = new OrderDate(frozenDate);

        // Assert.Equal(frozenDate, order.Date);
    }

    public void Dispose()
    {
        SystemTime.UnFreeze();
    }
}

This uses the constructor to freeze the SystemTime (class not shown) in order to test the constructor of the class OrderDate (also not shown). OrderDate uses SystemTime.Now internally.

After each test Fixie will take care of unfreezing the system time based on the logic in Dispose().

As you can see from the example, the test methods are about as clean as it's possible to get without not having tests. Your test class doesn't even need to reference Fixie, it's a blissful escape from the moral burden of writing tests because it's just writing code.

Another plus point is the massive array of configuration options if the defaults don't suit you. I have only skimmed the surface but there are full details in the documentation.

Undecided Points

At the time of writing Fixie doesn't run tests in parallel by default. For some use cases this is positive (e.g. integration tests) but some people may miss the parallel speed boost (not that I've found parallelism to ever deliver a speed boost).

You may also find yourself hunting around for the Assert.Equal() classes and methods you're used to from other test frameworks. Fixie doesn't include assertions because the library authors believe that the choice of assertion library is unrelated to the test framework.

After initially being bewildered by this I now simply install Shouldly, and feel like the exclusion of assertions is a Good Thing (TM); but your mileage may vary.

Extending It

This is where it transpires this entire post is an advertorial.

Currently Fixie doesn't have support for data driven tests out of the box. It's easy enough to set up and the documentation details it well but it's still a barrier for the lazy tester.

Data driven tests are where one test method runs multiple test cases by passing parameters into that method.

In an entirely artificial example let's say an OrderDate may be created 5 or more days in the future (but not between 1 to 4 days in the future).

Using xUnit style InlineData attributes we could test this as follows:

[InlineData(1)]
[InlineData(2)]    
[InlineData(3)]
[InlineData(4)]
public void OrderDateCannotBeLessThan5DaysInFuture(int daysInFuture)
{
    try
    {
        var order = new OrderDate(frozenDate.AddDays(daysInFuture));
    }
    catch (OrderDateException)
    {
        // Could run some assertion here if desired
    }
}

This will run 4 tests to check we cover the range we're expecting to be disallowed.

Maybe we also want to prevent a future date being a Saturday. You may think we could use the same in-line data attribute approach, e.g.:

[InlineData(new DateTime(2016, 03, 9))]

However C# attributes cannot contain DateTime or decimal data. This is due to Common Language Runtime (CLR) restrictions on how attributes are stored in the compiled code. This means we can't use our InlineData attribute for tests where the data isn't a primitive CLR type.

Fortunately xUnit provides the MemberData attribute for just this occasion. This attribute looks for a static method, property or field on a class based on the name of the member.

For our not-on-Saturdays test this might work as follows:

// The member used to provide test data
public static IEnumerable<object[]> SaturdayDates
{
    get
    {
        return new[]
        {
            new object[] {new DateTime(2016, 03, 5)},
            new object[] {new DateTime(2016, 03, 9)},
            new object[] {new DateTime(2016, 07, 9)}
        };
    }
}

// The test
[MemberData("SaturdayDates")]
public void OrderDateCannotBeOnSaturdays(DateTime saturdayDate)
{
    try
    {
        var order = new OrderDate(saturdayDate);
    }
    catch (OrderDateException)
    {
    }
}

Take my word for it that the Saturday dates in the property are all actually on Saturday.

This example will run 3 tests from the single test method to verify that a set of different Saturdays throws. You'd probably choose better edge cases, far into the future or closer to the SystemTime date, to check against.

I've carefully made a point of saying these are xUnit attributes, but luckily for you, dear reader, some kind soul has created the equivalent attributes in a handy NuGet package for Fixie.

To install from the package manager console use:

Install-Package UglyToad.Fixie.DataDriven

Or searching "fixie" in the NuGet package manager should return it.

Once it's installed you need to define a custom Fixie convention in order to use the attributes. This convention class can live anywhere in your test project. For example:

using Fixie;
using UglyToad.Fixie.DataDriven;

public class TestConfiguration : Convention
{
    public TestConfiguration()
    {
        Classes.NameEndsWith("Tests");

        Methods.Where(method => method.IsVoid());

        Parameters
            .Add<ProvideTestDataFromInlineData>()
            .Add<ProvideTestDataFromMemberData>();
    }
}

Then the attributes are available in your test project and will be detected by the test runner. The attributes are in the namespace UglyToad.Fixie.DataDriven.

Have fun redeeming your immortal programming soul with some tests!

Automated web UI testing using Selenium with Python

Eliot (Eliot Jones) — Sun, 29 Nov 2015 00:00:00 GMT

Legacy applications are a fact of life in software development. However well a piece of software starts and no matter how much care is taken to document it, as people leave and new people are assigned to look after it, knowledge about what the software does or how it's meant to work is lost.

For older code which was written before a time of TDD and testable architecture this creates a fear of change, both for a developer and for the business. Testing is time consuming and manual and the knowledge of what the expected inputs and outputs of the application are, are lost and changed as ownership of the application changes hands.

In his book Working Effectively with Legacy Code Michael Feathers outlines steps which can be taken inside the code to refactor it and start adding test coverage. Depending on the size of the codebase and the budgetary and time constraints it can be difficult to get very far with refactoring the codebase itself.

A useful tool

While automated tests should also be used for new software they can be just as, if not more, useful when it comes to legacy software. For example, the UI of a legacy application tends to be much more fixed than for a project currently undergoing development.

Automated UI tests for a web application can provide a useful time-saver and allow you to quickly cover the application with regression tests at a higher level.

One of the most popular tools for automated UI testing is Selenium. Selenium offers support for a range of languages, including .NET, Java, Ruby, etc. With Selenium you can quickly script interaction with different browsers, primarily Firefox but there are also drivers available for Internet Explorer, Google Chrome and Phantom.

A natural fit

While the concept of Business Driven Development (BDD) is orthogonal to automation testing, the two work well together.

Even though BDD is about more than just a diferent way of coding, the Gherkin style Given, When, Then syntax is a good fit for UI tests. Since these Gherkin requirements often represent the outcome the user is expecting to see when they use the application they tend to align more naturally with UI testing.

As an example of a business requirement written in the Gherkin style (I may have my Given and When the wrong way round in terms of the meaning of the steps):

Given a user requests access
When I am administrating the system
Then I can deny the access request

Using it

I've been playing around with the Python behave library and the Selenium bindings for Python. While I've written Selenium tests in .NET before this seems like overkill for what is effectively scripting a test suite. Python as a language seems a nicer, more lightweight, fit for Selenium with BDD and it ticks those nice open source boxes.

Since early project set-up can be an issue for adopting a UI testing approach I decided to make a reusable project template which can be used and extended for building UI tests. The nice thing about this project is as long as you have Python (3.5+) and a web browser on your system you can download it and have it running almost instantly.

This means you can start developing your features and steps without having to write any set-up code.

The project comes with one feature containing two demo scenarios:

Feature: Load Wikipedia home

Scenario: Load it
    Given I am not logged in
    When I am on the homepage
    Then I should see the English option

Scenario: Search Cats
    Given I am on the homepage
    When I search cats
    Then I should go to the cats page

All the plumbing responsible for changing browsers in the configuration and easily accessing your own additional configuration settings from code is taken care of.

As a result you can just write your features and steps without worrying about anything else. Here are the steps (except the "I am on the homepage" step which is in the shared steps) for the above feature:

import behave
from selenium import webdriver
from helpers import configuration
from selenium.webdriver.common.keys import Keys

@given("I am not logged in")
def step_impl(context):
    pass

@then('I should see the English option')
def step_impl(context):
    link = context.browser.find_element_by_link_text("English")
    assert True is True

@when("I search cats")
def step_impl(context):
    search = context.browser.find_element_by_id("searchInput")
    search.send_keys("cats")
    search.submit()

@then("I should go to the cats page")
def step_impl(context):
    title = context.browser.find_element_by_tag_name("h1")
    assert "Cat" in title.text

The project also offers:

Support for Internet Explorer, Chrome and Firefox
Screenshots on test failure
Configurable deletion of cookies to take place after Steps, Scenarios, Features or the entire test run
A single script to install all dependencies and create a isolated Virtual Environment to avoid polluting your global installation
Reporting of tests results in the JUnit format.

If you don't want to use Git you can just download the project as a zip and use it as the base for your development. Thanks to the install script it's also easy to integrate with Jenkins.

The best documentation for using the Selenium driver with Python is here.

Go give it a go and let me know how it goes :)

I'm hoping to extend it with support for Phantom as well as more utility functions for quickly selecting from a dropdown, etc.

PostHaste

Eliot (Eliot Jones) — Tue, 13 Oct 2015 00:00:00 GMT

I haven't blogged in a while, simply because I've not thought of any interesting topics to blog about, hopefully something will come along sooner or later!

In the meantime I'm promoting my little Visual Studio extension, PostHaste.

This extension aims to make sending code to hastebin (it's like a good version of PasteBin) from inside Visual Studio easier. Not that it could be much easier but it seemed like a fun way to learn Visual Studio extensibility.

It's (hopefully) compatible with Visual Studio 2013 and 2015, why not give it a try? :)

MVC postback for users without JavaScript - Post 2

Eliot (Eliot Jones) — Sun, 21 Jun 2015 00:00:00 GMT

Posts in this series:

Post 1
Post 2

So far we have created a form which performs a postback to the server to add more input fields to the same page. We have encountered an issue with MVC seeming to cache partial views (or at least the values inside them).

Step 4

Step 4 attempts to fix the caching of the selected value field.

I spent hours looking for a way to stop MVC caching partial views, which is what I thought was happening. I tried disabling the output cache and loading the partial views via actions, neither of which helped. I finally found this answer on StackOverflow; the ModelState holds the old values when a view is reloaded via an HTTP POST even if you change the values of the model in the action.

By removing the specific value from the ModelState the value clears properly and the view displays as expected.

private ActionResult AddPostback(OrderViewModel model)
{
    ModelState.Remove("SelectedMenuItem");
    model.SelectedMenuItem = null;

    return View("Order", model);
}

The form now works as expected, the "add another" button adds another input to the page by reloading the page, the submit button still submits the form to the server and redirects to the summary screen. Now we have the screen working for users without JavaScript however users get a "screen flicker" due to reloading the page when clicking "Add another".

See this image for an example of the screen flicker non-JavaScript users will see, this seems like an acceptable compromise, since there's no way to prevent the page having to reload for these users. However for people with JavaScript enabled it would be nice to offer a cleaner user experience.

Step 5

This step mainly focuses on using jQuery Ajax to submit the form in a user friendly way. I originally started using the Microsoft Ajax @Ajax.Helper() methods but it turned out to be easier to just write the JavaScript in this case.

The first main change is that the body of the form now lives in a new partial view (_OrderPartial) which contains the Current and Previous input partial views:

@model EliotJones.PostbackTutorial.ViewModels.Menu.OrderViewModel

@Html.ValidationSummary()
@Html.AntiForgeryToken()
@Html.Partial("_Previous", Model)
@Html.Partial("_Current", Model, ViewData)

The button for "Add another" in the _Current partial view now calls a JavaScript function using the onclick attribute:

<button type="submit" name="submit" value="add" onclick="addMore(event);">
    <span class="glyphicon glyphicon-plus"></span>
    Add another
</button>

Note that we pass event to the onclick attribute, this is required for the code to work on Firefox.

The code for this JavaScript function is in the Order view:

@model EliotJones.PostbackTutorial.ViewModels.Menu.OrderViewModel
<h2>Order</h2>
<hr/>

@using (Html.BeginForm("Order", "Menu", FormMethod.Post, htmlAttributes: new { id = "form" }))
{
    <div id="form-for-ajax">
        @Html.Partial("_OrderPartial", Model)
    </div>
    <button type="submit" name="submit" value="submit">
        Submit
    </button>
}

@section scripts{
    <script>
        function addMore(event) {
            // The code goes here
        }
    </script>
}

The code uses the jQuery Ajax post function:

function addMore(event) {
    $.post('@Url.Action("Order", new { submit = "add" })', $('#form').serialize())
        .done(function (data) {
            $('#form-for-ajax').html(data);
        }).fail(function () {
            return true;
        });

    event.preventDefault ? event.preventDefault() : event.returnValue = false;
}

The first parameter of the call to $.post() is a the URL to post to for this request. I've used the Razor helpers here to avoid hard-coding the entire URL. Since the value of submit will no longer come from the button we add it as a QueryString parameter:

$.post('@Url.Action("Order", new { submit = "add" })',

The single quotes ensure the URL helper value is a string in the JavaScript, miss them out at your peril.

The next argument to $.post() is the data to submit to the URL, this uses jQuery's Serialize to send the form data in the format required.

After a call to $.post(url, data) the code will continue to the next line, this is not the line you think it might be. Because the post runs asynchronously the next line which runs is:

event.preventDefault ? event.preventDefault() : event.returnValue = false;

This stops the browser submitting the form on click of the submit button in a similar way to return false;. This means for users with JavaScript this button will instead call an Ajax request whereas for users without the form will still submit (because none of this JavaScript code will run).

The done function does not run immediately because it is a "promise", which means it executes asynchronously. When the Ajax request completes successfully the code in done will run, alternatively if there's an error then fail will run.

$.post(url, submittedData)
    .done(function (data) {
        $('#form-for-ajax').html(data);
    })

The data passed to the function in done is the data returned by the request, in our case this is HTML, so we can just replace the content of the form with the returned data (using .html()) since the postback just returns the form with the extra input fields.

The problem now is that the controller code currently returns the entire form including layout again so we end up with nested forms:

Returning partial views when required

Luckily for us MVC 5 offers a convenience method for detecting Ajax requests:

if (Request.IsAjaxRequest())
{
    return PartialView("_OrderPartial", model);
}

This looks for the X-Requested-With: XMLHttpRequest header used by jQuery Ajax requests (mainly for security reasons, e.g. CSRF) and can be used to return the partial view for these JavaScript requests. This check is added in two places. See the full code for the finished Menu controller.

[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult Order(OrderViewModel model, string submit)
{
    if (model.SelectedMenuItem.HasValue)
    {
        model.SelectedItems.Add(model.SelectedMenuItem.Value);
    }

    BindSelectLists(model);

    if (!ModelState.IsValid || string.IsNullOrWhiteSpace(submit))
    {
        if (Request.IsAjaxRequest())
        {
            return PartialView("_OrderPartial", model);
        }
        return View(model);
    }

    switch (submit)
    {
        case AddValue:
            return AddPostback(model);
        default:
            return SubmitPostback(model);
    }
}

private ActionResult AddPostback(OrderViewModel model)
{
    ModelState.Remove("SelectedMenuItem");

    model.SelectedMenuItem = null;
    if (Request.IsAjaxRequest())
    {
        return PartialView("_OrderPartial", model);
    }
    return View("Order", model);
}

Summary

In summary:

Create the form and controller action with one submit button.
Add another submit button with a name and a different value.
You can use a parameter to the controller action with the name of the button to check the value of where the submit came from.
Return the correct ActionResult for your desired action, watching out for caching in the ModelState.
Add an Ajax request to the onclick of the submit button which calls the same controller action but returns partial views instead.

MVC postback for users without JavaScript - Post 1

Eliot (Eliot Jones) — Sat, 20 Jun 2015 00:00:00 GMT

Posts in this series:

Post 1
Post 2

Despite only around 1.2% of users having JavaScript disabled there is still a requirement to develop sites which work for users who have JavaScript turned off, especially sites which provide government services. The UK's Government Digital Services (GDS) require that a site works without JavaScript and is then enhanced using JavaScript. This causes all sorts of problems designing forms, especially where there is an "Add more" option for input fields.

Designing something like an "Add more" functionality with JavaScript is fairly trivial, having it work for those without is less so. Having encountered this problem several times I've written a summary of the approach I have used to address this.

The code

All the code for this tutorial is available on GitHub. I have used Git's branches to take snapshots of development changes. The final code is in the branch "step-5". Additonally I have hosted the website on Azure so you can see what the final page does here.

The product

We are designing a page to allow users to choose items from a menu, once they have selected as many items as they want, they submit the form and are taken to a summary of their active orders:

Getting started

The postback concept is from ASP.NET Webforms and there's no real parallel in MVC, which is a good thing because the postback model encouraged ignorance of a site working with HTTP in a stateless manner. However it is still possible to create a postback style model in MVC.

We start with a simple almost empty MVC web application project with a Home controller and a Menu controller:

public class MenuController : Controller
{
    [HttpGet]
    public ActionResult Order()
    {
        return View();
    }
}

There's also a Meal model in the Models folder:

public class Meal
{
    public int Id { get; set; }

    public decimal Price { get; set; }

    public string Name { get; set; }

    public int NumberRemaining { get; set; }

    public decimal TimeToServe { get; set; }
}

And a class called Menu which provides a list of different meals and the ability to order them.

Step 1

The first step is to create a form to allow the user to select one meal and post to the server. We use a ViewModel for data binding to our view:

public class OrderViewModel
{
    public SelectList MenuItems { get; set; }

    [Required(ErrorMessage = "Please select a menu item.")]
    public int? SelectedMenuItem { get; set; }

    public IList<int> SelectedItems { get; set; }

    public OrderViewModel()
    {
        this.SelectedItems = new List<int>();
    }
}

This has a corresponding Get and Post action on the [controller] and the following view:

@{
    ViewBag.Title = "Order";
}

<h2>Order</h2>

@using (Html.BeginForm())
{
    @Html.ValidationSummary()

    <div class="form-group">
        @Html.LabelFor(m => m.SelectedMenuItem, "Select a menu item")
        @Html.ValidationMessageFor(m => m.SelectedMenuItem)
        @Html.DropDownListFor(m => m.SelectedMenuItem, Model.MenuItems, "Please select", new {@class = "form-control"})
    </div>
    <button type="submit">Submit</button>
}

Currently this view has one submit button which posts the entire form.

Step 2

The next step adds a second submit button to the view:

<div class="form-group">
    @Html.LabelFor(m => m.SelectedMenuItem, "Select a menu item", new { @class = "dropdown-label" })
    @Html.ValidationMessageFor(m => m.SelectedMenuItem)
    @Html.DropDownListFor(m => m.SelectedMenuItem, Model.MenuItems, "Please select", new {@class = "form-control dropdown"})
    <button type="submit" name="submit" value="add"><span class="glyphicon glyphicon-plus"></span>Add another</button>
</div>
<button type="submit" name="submit" value="submit">Submit</button>

Both buttons now have a name attribute (submit) and different value attributes, "add" and "submit" respectively.

We can now detect which of the two buttons have been clicked in the controller action and do different things for either button. Currently our controller submits the form for both buttons.

Step 3

Next we set the controller to treat a post sent using the add button differently to one sent using submit:

[HttpPost]
public ActionResult Order(OrderViewModel model, string submit)
{
    if (model.SelectedMenuItem.HasValue)
    {
        model.SelectedItems.Add(model.SelectedMenuItem.Value);
    }

    BindSelectLists(model);

    if (!ModelState.IsValid || string.IsNullOrWhiteSpace(submit))
    {
        return View(model);
    }

    switch (submit)
    {
        case AddValue:
            return AddPostback(model);
        default:
            return SubmitPostback(model);
    }
}

The controller action takes a string parameter with the same name as the submit buttons, the value of this will either be "add" or "submit", the values we set for the buttons in our HTML.

The action first adds the currently selected item to the list of previously selected items if present. After checking that the ModelState is valid (i.e. that there is a currently selected item) it chooses one of two methods to call depending on the button pressed. There is a method for the add post and the submit post.

public ActionResult AddPostback(OrderViewModel model)
{
    model.SelectedMenuItem = null;

    return View("Order", model);
}

public ActionResult SubmitPostback(OrderViewModel model)
{
    foreach (var id in model.SelectedItems)
    {
        Meal meal = Menu.Single(m => m.Id == id);

        Menu.Order(meal);
    }

    return RedirectToAction("Summary");
}

Add: Sets the currently selected item to null and returns the view.
Submit: Orders the items and moves to the summary screen.

It's important to note that this approach goes against the way the MVC framework is designed, by design MVC treats the return of a view (rather than a redirect) from an HTTP POST as being due to Validation errors.

In this step there have been some changes to the view to provide partial views showing the current selection input and previously selected items.

Currently we have a problem, when we click "Add another" and the page performs a postback, the current input still loads with the previous value selected. Even though we set the selected value to null in the AddPostback method and it loads in the view as null it's not resetting in the view. Somehow MVC has cached the value for this dropdown.

We will address this problem in the next post.

Core CLR and System.Xml

Eliot (Eliot Jones) — Tue, 09 Jun 2015 00:00:00 GMT

If you need to use some package which is dependent on System.Xml with Core CLR / DNX Core you simply add the Nuget package System.Xml.XmlDocument rather than referencing the GAC System.Xml.

This allows you to keep your code in Core CLR friendly mode.

Visual Studio 2015 and Gulp

Eliot (Eliot Jones) — Wed, 03 Jun 2015 00:00:00 GMT

Having played around with the Visual Studio 2015 preview a bit I've tried to learn how Gulp integrates with the Web Application project.

Gulp seems like one of those many, many, constantly appearing JavaScript libraries with names like YamahaPotplantSaucepan.js which are impossible to keep up with. However since it takes the place of the Bundle config it's important to be able to achieve a basic level of familiarity with it.

What does it do?

Gulp describes itself as a "streaming build system" which basically means you can wire up a pipeline of operations to perform on your content, such as JavaScript and CSS.

By default the project template comes with a gulpfile (shown in the image as #1). This is configured to take the dependencies from Bower (which is a package manager loading a few jQuery packages in by default) and copy them to the wwwroot/lib folder (#3 in the image).

The Bower packages are listed in the bower.json file.

The standard gulpfile looks like this:

/// <binding Clean='clean' />

var gulp = require("gulp"),
        rimraf = require("rimraf"),
        fs = require("fs");


eval("var project = " + fs.readFileSync("./project.json"));

var paths = {
    bower: "./bower_components/",
    lib: "./" + project.webroot + "/lib/"
};

gulp.task("clean", function (cb) {
    rimraf(paths.lib, cb);
});

gulp.task("copy", ["clean"], function () {
    var bower = {
        "bootstrap": "bootstrap/dist/**/*.{js,map,css,ttf,svg,woff,eot}",
        "bootstrap-touch-carousel": "bootstrap-touch-carousel/dist/**/*.{js,css}",
        "hammer.js": "hammer.js/hammer*.{js,map}",
        "jquery": "jquery/jquery*.{js,map}",
        "jquery-validation": "jquery-validation/jquery.validate.js",
        "jquery-validation-unobtrusive": "jquery-validation-unobtrusive/jquery.validate.unobtrusive.js",
        "jquery-ui": "jquery-ui/*.{js,map}"
    }

    for (var destinationDir in bower) {
        gulp.src(paths.bower + bower[destinationDir])
          .pipe(gulp.dest(paths.lib + destinationDir));
    }
});

This script defines 2 Gulp tasks:

clean - uses a force clean to delete everything in the wwwroot/lib folder (rimraf is a npm package which recursively force deletes all files in the target folder, equivalent to the rm -rf command).
copy - runs a clean first and then copies the scripts from their Bower package location to the wwwroot/lib/NAME folder, where NAME is the left hand of the bower json object.

How do I run it?

The Gulp tasks can be run manually from the Task Runner Explorer (if you can't see it go to Views > Other Windows > Task Runner Explorer). The result of running the copy task is shown below:

The left hand side has the 2 tasks listed, clean and copy. Right click a task and click "Run" to run it. Tasks can also be bound to some defined bindings, for example you may want to copy on every build:

How do I modify it?

In order to use our own files with Gulp we need to modify the Gulp task or write a new one. There are good guides on using Gulp to build a pipeline. This blog post isn't one of them.

We have a folder we use to write all our custom Javascript in, for this tutorial this is in ProjectRoot/Scripts. First we can define this path as a JavaScript variable, I stole space in the pre-defined paths object:

var paths = {
    bower: "./bower_components/",
    lib: "./" + project.webroot + "/lib/",
    scripts: "./Scripts/"
};

The . is the project root.

Since I don't want to be running multiple tasks where I only need one I'm going to put my code in the copy task:

gulp.task("copy", ["clean"], function () {
    var bower = {
        "bootstrap": "bootstrap/dist/**/*.{js,map,css,ttf,svg,woff,eot}",
        // The rest of the bower paths
    }

    for (var destinationDir in bower) {
        gulp.src(paths.bower + bower[destinationDir])
          .pipe(gulp.dest(paths.lib + destinationDir));
    }

    // Copy all project scripts into the lib folder.
    gulp.src(paths.scripts + '**.js')
        .pipe(gulp.dest(paths.lib));
});

This takes all my scripts from the source (./Scripts folder), it then pipes them into the destination operation which writes them into the ./wwwroot/lib folder.

Note - if the scripts don't show up, look at the lib folder in the Windows File Explorer; Visual Studio sometimes doesn't show the copied Gulp files.

Now we'd like to minify these files, luckily for us we can add a step to our streaming operation.

First we need a component to do the minification. We'll use the node package gulp-uglify to do this. To install this package open the Package.json folder (#2 from the first picture). Put a line in there to reference the uglify package:

{
  "name": "ASP.NET",
  "version": "0.0.0",
  "devDependencies": {
    "gulp": "3.8.11",
    "rimraf": "2.2.8",
    "gulp-uglify": "1.2.0"
  }
}

This will cause the node package manager to automatically restore these dependencies. We can now use this in our gulpfile. Modify the top of the gulpfile to include the uglify component:

var gulp = require("gulp"),
        rimraf = require("rimraf"),
        fs = require("fs"),
        uglify = require("gulp-uglify");

Finally you can change the custom code in the copy task to add an Uglify step to the pipeline:

// Copy all project scripts into the lib folder.
gulp.src(paths.scripts + '**.js')
    .pipe(uglify())
    .pipe(gulp.dest(paths.lib));

SpecFlow looking for When instead of Given step

Eliot (Eliot Jones) — Wed, 20 May 2015 00:00:00 GMT

When writing some automation tests using Selenium with SpecFlow recently I was faced with an odd error:

    No matching step definition found for the step. Use the following code to create one:
    [When(@"I add a new dog")]
    public void WhenIAddANewDog()
    {
        ScenarioContext.Current.Pending();
    }

The suggested code was using a [When("My Scenario")] step despite the feature file declaring it as a Given:

Scenario: Check for existing dog
    Given I have a new Dog Controller
    And I add a new dog
    When I check if the added dog exists
    Then The check is true

(This feature is written badly to illustrate the error).

The step it couldn't find was the And I add a new dog step. Despite showing as bound in the feature file and being able to navigate to the step definition, the running test couldn't find it and was looking for a When instead of a Given.

This is because the previous step was calling sub steps as follows:

public class DogControllerSteps : Steps
{
    [Given("I have a new Dog Controller")]
    public void CreateDogController()
    {
        Given("I have a new query bus");

        When("I create a new dog controller");
    }
}

Inheriting from Steps allows us to reuse step definitions from the same or other step files.

When SpecFlow runs this it knows that it last ran a When step, despite being in the definition of a Given. Therefore when the next step is defined with And, it looks for a When. To fix this you can simply change your feature to be more specific:

Scenario: Check for existing dog
    Given I have a new Dog Controller
    Given I add a new dog
    When I check if the added dog exists
    Then The check is true

Passing tests :)

3-tier CQRS using Web API as a command/query bus

Eliot (Eliot Jones) — Sat, 16 May 2015 00:00:00 GMT

We recently started using CQRS on a project to provide a new web application to manage a complicated permit process. The CQRS pattern combined with a proper domain model seemed like a good fit for the business logic.

Midway through development of our 2 tier solution (Web <-> Database) we were asked to instead asked to develop a 3 tier solution (Web <-> Api <-> Database) to avoid exposing the database to the web server (no WCF allowed).

Initially the change slowed development and resulted in an explosion in the Lines of Code to functionality ratio. After some thinking we realised it might be possible to use Web API as a command/query bus for our pre-existing CQRS setup, effectively allowing us to "pass-through" the API without needing to write a controller action per command.

This post summarises the ideas behind the working approach we developed. All the code used in this post can be found on Github but the code is very obviously simplified; both to avoid revealing implementation specific details such as security and to allow this post to focus on the concepts.

CQRS

If you're not familiar with CQRS read some of the articles available on the advantages and disadvantages of the pattern. Basically you separate commands (actions which update data) from queries (which read data).

This post will only deal with implementing the query side of CQRS over Web API. Commands should be simpler.

Query and QueryHandler interfaces are defined in a separate CQRS class library (ideally Queries and Query handlers live in separate libraries because your web project shouldn't have to reference query handlers).

public interface IQuery<TReturn>
{
}

public interface IQueryHandler<TQuery, TReturn> where TQuery : IQuery<TReturn>
{
    Task<TReturn> Send(TQuery query);
}

All aboard the query bus

We generally want to use our bus as a mediator so we don't have to specifically tie queries and handlers together. The idealised implementation is:

TQueryResult result = await queryBus.SendAsync(new Query());

The bus should find and call the correct handler for the query and return the result. In this implementation the Web API controller action replaces the bus.

public class QueryController : ApiController
{
    private readonly IServiceLocator serviceLocator;

    public QueryController()
    {
        serviceLocator = new NaiveLocator();
    }

    public async Task<object> Send(QueryRequest query)
    {
        // TODO: Implement the quey bus, find the right handler for the query.
    }
}

I haven't wired up a dependency injection container in this example so created a very simple service locator (anti-pattern!):

internal interface IServiceLocator
{
    object Resolve(Type serviceType);
}

The implementation of this locator takes the type of QueryHandler to find and returns that handler. In a real example this could be replaced with a dependency injection container's resolve method:

internal class NaiveLocator : IServiceLocator
{
    private readonly Dictionary<Type, object> services; 

    public NaiveLocator()
    {
        this.services = new Dictionary<Type, object>();

        services.Add(typeof(IQueryHandler<GetAllRappers, Rapper[]>), new GetAllRappersHandler(new Data()));
        services.Add(typeof(IQueryHandler<GetRapperByName, Rapper>), new GetRapperByNameHandler(new Data()));
    }

    public object Resolve(Type serviceType)
    {
        return services[serviceType];
    }
}

Getting on the bus

Because we are sending our queries as HTTP requests to our API the queries have to be serialized somehow and all generic type parameter information is lost.

One approach is to wrap the query as Json in a parent Json object which also contains type information:

public class QueryRequest
{
    public string QueryTypeName { get; set; }

    public string QueryData { get; set; }
}

When the website sends the request it has to generate this object (this method is in ApiClient in the Api.Client project):

public async Task<TResult> Send<TQuery, TResult>(TQuery query) where TQuery : IQuery<TResult>
{
    var request = new QueryRequest
    {
        QueryData = JsonConvert.SerializeObject(query),
        QueryTypeName = typeof(TQuery).AssemblyQualifiedName
    };

    var response = await client.PostAsJsonAsync("api/Query/Send", request);

    return await response.Content.ReadAsAsync<TResult>();
}

It's important to use the AssemblyQualifiedName rather than FullName or just Name.

The wheels on the bus

When our QueryRequest object arrives the API knows nothing about it. The first thing to do is work out what the query is and what we're expected to return. In our API controller we construct an object representing what we know about the request:

var typeInformation = new QueryRequestTypeInformation(query);

The constructor for QueryRequestTypeInformation looks like this:

internal class QueryRequestTypeInformation
{
    public Type QueryType { get; private set; }

    public Type ResultType { get; private set; }

    public QueryRequestTypeInformation(QueryRequest query)
    {
        this.QueryType = Type.GetType(query.QueryTypeName);
        this.ResultType = this.QueryType.GetInterfaces().Single(i => i.Name.Contains("IQuery")).GetGenericArguments()[0];
    }
}

This object will now contain information about the query type and the result type.

We can then use this to work out the type of handler we need our Service Locator to locate (in the API controller):

var queryObject = JsonConvert.DeserializeObject(query.QueryData, typeInformation.QueryType);

var handlerType = typeof(IQueryHandler<,>).MakeGenericType(typeInformation.QueryType, typeInformation.ResultType);

Jonas Gauffin's CQS library provided the inspiration for much of the controller code.

Given we now have the type of the handler we need, we can request it from our service locator (yes, I'm trying to mention the name of that particular anti-pattern enough times to drive home the point that this isn't the best code, but it works):

var handler = serviceLocator.Resolve(handlerType);

A moment of reflection

The rest of the code uses reflection heavily, of course there are performance penalties associated with the use of reflection. So far this seems negligible compared to the time for 2 web request/responses (client -> web -> api -> web -> client) and the data access time.

To call the right handler for our query and return an object the API controller then does this:

var sendMethodInfo = handlerType.GetMethod("Send");

return await (dynamic)sendMethodInfo.Invoke(handler, new[] { queryObject });

I wasn't aware that dynamics could be awaited in this manner until reading a comment from Stephen Cleary on my Stack Overflow question on the topic.

The full API controller code then looks like this:

public async Task<object> Send(QueryRequest query)
{
    var typeInformation = new QueryRequestTypeInformation(query);

    var queryObject = JsonConvert.DeserializeObject(query.QueryData, typeInformation.QueryType);

    var handlerType = typeof(IQueryHandler<,>).MakeGenericType(typeInformation.QueryType, typeInformation.ResultType);

    var handler = serviceLocator.Resolve(handlerType);

    var sendMethodInfo = handlerType.GetMethod("Send");

    return await (dynamic)sendMethodInfo.Invoke(handler, new[] { queryObject });
}

Summary

As mentioned in the introduction this is not a complete sample and should not be used in production! For a start there's no security at all between the API and web layer. Additionally there's no error handling anywhere.

This post outlines a possible approach to a generic endpoint for a Web API.

Full code with working sample.

bus: https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/ICCE_Fist_Student_Wallkill_bus.JPG/640px-ICCE_Fist_Student_Wallkill_bus.JPG

Fun with Nuget

Eliot (Eliot Jones) — Tue, 14 Apr 2015 00:00:00 GMT

I recently finished the first release of my 2nd Nuget package which provides a class to convert an ADO.Net DataTable to a list of objects. This is the result of an effort to create a library for the conversion technique outlined in this post.

I've also created a Nuget package to help out with MVC 5 Radio Buttons located here.

This post summarises some things I learned from creating these Nuget packages.

Simplest package

For simply packing a class library targeting a single version of the .Net framework you can simply invoke the Nuget.exe with the path to the .csproj:

.\.nuget\NuGet.exe pack .\Path\To\Project.csproj -Prop Configuration=Release

If your solution doesn't have a copy of the Nuget exe in the .nuget solution folder you can download the exe separately.

This command will create the nuget package in the current directory with the metadata declared in the AssemblyInfo.cs for the project.

Creating a specification

The default nuspec specification file created by calling:

Nuget.exe spec

Is fairly self explanatory:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>Package</id>
    <version>1.0.0</version>
    <authors>Eliot</authors>
    <owners>Eliot</owners>
    <licenseUrl>http://LICENSE_URL_HERE_OR_DELETE_THIS_LINE</licenseUrl>
    <projectUrl>http://PROJECT_URL_HERE_OR_DELETE_THIS_LINE</projectUrl>
    <iconUrl>http://ICON_URL_HERE_OR_DELETE_THIS_LINE</iconUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Package description</description>
    <releaseNotes>Summary of changes made in this release of the package.</releaseNotes>
    <copyright>Copyright 2015</copyright>
    <tags>Tag1 Tag2</tags>
    <dependencies>
      <dependency id="SampleDependency" version="1.0" />
    </dependencies>
  </metadata>
</package>

If you have several dlls which target different versions of the .Net framework (for example) you can declare the mapping of files to the package contents like this:

<?xml version="1.0"?>
<package >
  <metadata>
    <id>Package</id>
    <version>1.0.0</version>
    <authors>Eliot</authors>
    <owners>Eliot</owners>
    <licenseUrl>http://LICENSE_URL_HERE_OR_DELETE_THIS_LINE</licenseUrl>
    <projectUrl>http://PROJECT_URL_HERE_OR_DELETE_THIS_LINE</projectUrl>
    <iconUrl>http://ICON_URL_HERE_OR_DELETE_THIS_LINE</iconUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Package description</description>
    <releaseNotes>Summary of changes made in this release of the package.</releaseNotes>
    <copyright>Copyright 2015</copyright>
    <tags>Tag1 Tag2</tags>
    <dependencies>
      <dependency id="SampleDependency" version="1.0" />
    </dependencies>
  </metadata>
    <files>
        <file src="Project\bin\Release\Project.dll" target="lib\net45"/>
        <file src="Project\bin\Release\Project.dll" target="lib\net40"/>
    </files>
</package>

Additionally content you want to install into the user's project on installation of your package can be declared in the files area:

<files>
    <file src="..\MvcProject\Views\Shared\EditorTemplates\*.cshtml" target="Content\Views\Shared\EditorTemplates\"/>
    <file src="..\OtherProject\bin\Release\OtherProject.dll" target="lib\net45"/>
</files>

Targeting multiple versions of the .Net framework

One thing I wanted to do with the DataTable converter project was allow users of .Net 3.5, 4.0 and 4.5 to install and use the converter. Assuming you have a project currently targeting .Net 4.5 that you want to provide compatible versions of for other frameworks you can create a new project with the correct Target Framework, e.g. My.Project and My.Project.Net35.

To avoid having multiple versions of the same code you can then add the code from your main version of the library as linked files.

For example with the following project and new Net 3.5 version:

You can add the classes already defined by right clicking the project and choosing Add -> Existing Item:

Select the files from your main project and then before clicking "Add", use the arrow to the right of the button to Add as link:

Edits to the linked file will also change the files in the main project so if you need to make changes for compatibility you'll have to create a copy of those files.

Entity Framework 6 Enums with String column

Eliot (Eliot Jones) — Mon, 23 Mar 2015 00:00:00 GMT

I recently watched Jimmy Bogard's excellent NDC presentation on Domain models to learn about how to implement a real world domain model.

One of the things Jimmy does in his presentation is to refactor his enum to use a base class allowing far more expressive enums with custom behaviour

There are a few versions of this custom enum base class around but I'm going to be using this one.

Annoyingly Entity Framework (EF) doesn't support mapping database fields to enums unless you use an int column in your database. This might be good enough for your implementation however it seems a bit fragile to couple your database to a mystery int defined in code somewhere which could be changed by any user who doesn't realise what they're changing.

For a marginally less fragile and more domain model friendly enum I used an adapted version of Jimmy's enumeration class:

The main change is to the DisplayName property. We add a setter because Entity Framework is only going to bring back the string display name and we need to map to the underlying value field to get the proper enum.

public string DisplayName
{
    get
    {
        return this.displayName;
    }

    // Entity Framework will only retrieve and set the display name.
    // Use this setter to find the corresponding value as defined in the static fields.
    protected set
    {
        this.displayName = value;

        // Get the static fields on the inheriting type.
        foreach (var field in GetType().GetFields(BindingFlags.Public | BindingFlags.Static))
        {
            // If the static field is an Enumeration type.
            var enumeration = field.GetValue(this) as Enumeration;
            if (enumeration == null)
            {
                continue;
            }

            // Set the value of this instance to the value of the corresponding static type.
            if (string.Compare(enumeration.DisplayName, value, true) == 0)
            {
                this.value = enumeration.Value;
                break;
            }
        }
    }
}

This simply checks the static fields on the instance of the enumeration class and finds the enumeration with the matching name.

We also change the private displayName and value fields to remove the readonly access modifier (and the underscores because I've never been able to adapt to the convention):

private int value;
private string displayName;

I also overrode the == reference equals check to make our enumeration class behaves more like a value type:

// Override reference equals to provide a value equals since the type is (almost) immutable.
public static bool operator ==(Enumeration a, Enumeration b)
{
    if (object.ReferenceEquals(a, b))
    {
        return true;
    }

    if ((object)a == null || (object)b == null)
    {
        return false;
    }

    var typeMatches = a.GetType().Equals(b.GetType());
    var valueMatches = a.Value.Equals(b.Value);

    return typeMatches && valueMatches;
}

public static bool operator !=(Enumeration a, Enumeration b)
{
    return !(a == b);
}

We need an additional step for our model configuration. Imagine we're storing a record of our prized Bonsai tree collection:

public class BonsaiTree
{
    public int Id { get; protected set; }

    public string Name { get; protected set; }

    public Genus Genus { get; protected set; }
}

public class Genus : Enumeration
{
    public static readonly Genus Leafy = new Genus(0, "Leafy");

    public static readonly Genus Twiggy = new Genus(1, "Twiggy");

    private Genus() { }

    private Genus(int value, string displayName) : base(value, displayName) { }
}

Where Bonsai tree is our EF entity. We need to tell EF that Genus is a complex type so it will map it to the same table. Additionally we need to ignore the value which we are not storing (if you needed to you could by extending this approach). Finally we need to configure the correct column name for the DisplayName of Genus:

public class EnumTestContext : DbContext
{
    public DbSet<BonsaiTree> BonsaiTrees { get; set; }

    protected override void OnModelCreating(DbModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.ComplexType<Genus>().Ignore(bt => bt.Value);
        modelBuilder.Types<BonsaiTree>().Configure(t => t.Property(bt => bt.Genus.DisplayName).HasColumnName("Genus"));
    }
}

I've only used this approach with Code First against an existing database so I'm not sure the migrations would handle it properly and I haven't tested it extensively however in a quick test it seemed to work well. Also the more derived Enum types for proper logic separation as detailed in the end of this post won't work because of how EF handles (or rather doesn't handle) inheritance in its mapping.

The full code is here; sorry about half the namespace disappearing, I was trying desperately to get PHP to stop displaying the Byte Order Mark and instead succeeded in getting it to eat the code.

FakeItEasy Heisenbugs

Eliot (Eliot Jones) — Tue, 17 Mar 2015 00:00:00 GMT

Update: the problem has been solved in version 2.0.0 of FakeItEasy.

I was recently facing an issue where my FakeItEasy tests were sometimes randomly failing. I'm specifying that it was the FakeItEasy tests because I managed to isolate to the failures to 2 tests which were using:

var thing = A.Fake<IInterface>();

The FakeItEasy fake would sometimes ignore the specified setup, such as:

A.CallTo(() => thing.Method()).Returns(true);

The fake would return false or null for reference types.

Analysing the stack trace on the failing test it was due to aggregate Exceptions occurring when running the tests in parallel. There is an issue describing this here.

We were using xUnit as our testing framework which runs tests in parallel by default. Unfortunately both FakeItEasy and Moq aren't threadsafe and this introduces bugs when tests are run in parallel. It seems like other testing frameworks also experience the same problem.

To get around this we disabled parallel test execution in xUnit. This means our tests take longer but are reliable.

If you've run into this issue before and found a better fix I'd be intrigued to hear it.

Dependency Injection Containers - Cheat Sheet

Eliot (Eliot Jones) — Sun, 01 Feb 2015 00:00:00 GMT

There are a lot of articles out there that describe both Dependency Injection and Inversion of Control. The purpose of this one is to serve as a quick reference for the things I keep forgetting because configuring a DI container is something I do so infrequently. This is a very simple guide showing the easiest scenarios and the simplest syntax. It walks through the same 3 steps:

1. Create Container
2. Configure Container
3. Retrieve from Container

In the following DI libraries:

Ninject
Unity
SimpleInjector
StructureMap

Of course it will feature examples tenuously related to animals.

Principals

A Dependency Injection library makes dependency injection easier by doing the work of wiring up an "object graph" (collection of related objects) for your application to use. This is generally done at the entry point to an application:

For console apps this is in the main method of Program.cs.
For MVC and Web API apps this is in Startup.cs.
For class libraries it is difficult to include dependency injection in the library itself as these have no defined entry point. One approach is to rely on the calling application to wire up the dependencies.
For WebForms apps it is possible to setup the DI container in the Global.asax Application_Start method however it's a bit more complicated.

Console App Example

In this example we're going to begin building a console app which simulates an ecosystem of toads.

The main logic of our simulation will run in the ToadSimulator Class:

public class ToadSimulator
{
    private readonly ILogger logger;
    private readonly SimulationSettingsProvider settingsProvider;
    private ICollection<Toad> simulationToads;

    public ToadSimulator(ILogger logger, SimulationSettingsProvider settingsProvider)
    {
        this.logger = logger;
        this.settingsProvider = settingsProvider;
    }

    public void InitialiseSimulation()
    {
        simulationToads = settingsProvider.RetrieveSimulationToads();

        logger.LogMessage(string.Format("ToadSimulator - Retrieved {0} toads to start the simulation.", 
                            simulationToads.Count));
    }
}

This is written to allow constructor injection of both the logger and the class which provides the settings for the simulation.

The settings provider also has a dependency on a logger:

public class SimulationSettingsProvider
{
    private readonly ILogger logger;

    public SimulationSettingsProvider(ILogger logger)
    {
        this.logger = logger;
    }

    internal virtual ICollection<Toad> RetrieveSimulationToads()
    {
        logger.LogMessage("SimulationSettingsProvider - Retrieving Toads.");

        return Database.RetrieveToads();
    }
}

We also provide an implementation of the logger interface:

public interface ILogger
{
    void LogMessage(string message);
}

public class ConsoleLogger : ILogger
{
    private string logBuffer;

    public ConsoleLogger()
    {
        logBuffer = string.Format("Logger (Id = {0}):", this.GetHashCode());
    }

    public void LogMessage(string message)
    {
        logBuffer += Environment.NewLine + message;

        Console.WriteLine(logBuffer);
    }
}

The console logger contains a per instance string so we can see exactly where the logger is getting called from and this will help us to understand the difference between object scopes later. It also records its hashcode (uniquely identifying integer) as an extra tool to understand object scope.

Finally all this gets invoked from the main method of the program:

public static void Main(string[] args)
{
    ToadSimulator toadSimulator = new ToadSimulator(new ConsoleLogger(), 
        new SimulationSettingsProvider(new ConsoleLogger()));

    toadSimulator.InitialiseSimulation();
}

Apart from the code in the main method the rest of the classes will not change from now on.

Even with this very simple example we can see that manually creating the dependencies to inject through the code is going to become a lot of work.

This is why we turn to a dependency injection library.

Resolve Using Ninject

Ninject can be added to a project through NuGet.:

Ninject uses a kernel to store its dependencies, we create one of these in our main method as follows:

IKernel kernel = new StandardKernel();

We now need to bind our logger interface to an implementation. This is done in the configuration of our container. It's useful to have a separate method in another class to configure our container:

internal class Bootstrapper
{
    public static void ConfigureContainer(IKernel kernel)
    {
        kernel.Bind<ILogger>().To<ConsoleLogger>();
    }
}

Once we have resolved the interface we get the kernel to provide us the class we need in our application by using the Get method:

public static void Main(string[] args)
{
    IKernel kernel = new StandardKernel(); // Create Container

    Bootstrapper.ConfigureContainer(kernel); // Configure Container

    ToadSimulator toadSimulator = kernel.Get<ToadSimulator>(); // Retrieve from Container

    toadSimulator.InitialiseSimulation();

    Console.ReadKey();
}

The output of this shows us that like in the original version, without the container, the logger is instantiated twice (the hashcodes differ):

Scope Using Ninject

We can change to scope of our logger as follows:

kernel.Bind<ILogger>().To<ConsoleLogger>().InTransientScope();

This is the default scope of Ninject bindings and is the same as leaving the InTransientScope off. An alternative scope is:

kernel.Bind<ILogger>().To<ConsoleLogger>().InSingletonScope();

With the output showing the logger is a singleton:

Convention Binding Using Ninject

Where we have lots of interfaces or abstract classes to bind, the binding code can quickly become repetitive. Instead we can use convention based configuration. To do this in Ninject we need to add another reference from NuGet. Install the ninject.extensions.conventions package. The kernel can now be bound using lambda expressions.

This example binds interfaces to their only implementing class, obviously this won't work where there are multiple interface implementations:

kernel.Bind(configure => configure.FromThisAssembly()
                .SelectAllClasses().BindDefaultInterfaces());

Using Unity

There are a huge range of DI Libraries, I'm just going to record the example above with a few other libraries, I'm getting these from NuGet. Firstly Microsoft's Unity:

public static void Main(string[] args)
{
    IUnityContainer container = new UnityContainer(); // Create Container

    Bootstrapper.ConfigureContainer(container); // Configure Container

    ToadSimulator toadSimulator = container.Resolve<ToadSimulator>(); // Retrieve from Container

    toadSimulator.InitialiseSimulation();

    Console.ReadKey();
}

And the Bootstrapper code:

public static void ConfigureContainer(IUnityContainer container)
{
    container.RegisterType<ILogger, ConsoleLogger>();
}

With the scope changed:

container.RegisterType<ILogger, ConsoleLogger>(new ContainerControlledLifetimeManager());

The Unity docs located here are great and go through using Unity with Convention based configuration and using Unity with WCF, Webforms and other more advanced use cases.

Using SimpleInjector

Creating a container and retrieving instances:

public static void Main(string[] args)
{
    Container container = new Container(); // Create Container

    Bootstrapper.ConfigureContainer(container); // Configure Container

    ToadSimulator toadSimulator = container.GetInstance<ToadSimulator>(); // Retrieve from Container

    toadSimulator.InitialiseSimulation();

    Console.ReadKey();
}

Bootstrapper code:

public static void ConfigureContainer(Container container)
{
    container.Register<ILogger, ConsoleLogger>();
}

With the scope changed:

container.RegisterSingle<ILogger, ConsoleLogger>();

The SimpleInjector docs located here are very much improved from before and are easy to follow.

Using StructureMap

Creating a container and retrieving instances:

public static void Main(string[] args)
{
    IContainer container = Bootstrapper.ConfigureContainer(); // Configure Container

    ToadSimulator toadSimulator = container.GetInstance<ToadSimulator>(); // Retrieve from Container

    toadSimulator.InitialiseSimulation();

    Console.ReadKey();
}

Bootstrapper code:

public static IContainer ConfigureContainer()
{
    return new Container(x => x.For<ILogger>().Use<ConsoleLogger>()); // Create Container
}

This registers the logger as a singleton instance. For a transient scope the below code worked:

return new Container(x => x.For<ILogger>().Use<ConsoleLogger>().AlwaysUnique());

The StructureMap documentation here is the easiest to follow.

Use with Web API and MVC

Most DI libraries provide a github package specifically tailored to setting up DI in MVC and/or Web Api. For example Unity provides the classes required to call the following method from the WebApiConfig:

public static void Register(HttpConfiguration config)
{
    // Web API configuration and services
    UnityWebApiActivator.Start();
}

Summary

This guide shows how the syntax changes between different DI libraries but the concepts remain the same. It does not provide any detail on advanced usage of DI libraries however should hopefully provide a quick start when changing between DI library syntax.

EF7 Table Mapping Exception

Eliot (Eliot Jones) — Thu, 15 Jan 2015 00:00:00 GMT

Note: This blog post relates to a library undergoing development and as such the information is likely to become outdated.

Even with Database First through the EDMX gone in Entity Framework 7 it's still possible to work with existing databases.

While trying this out with one of my databases I ran into the following Exception:

<Message>An error has occurred.</Message>

<ExceptionMessage>Invalid object name 'SomeClass'.</ExceptionMessage>

<ExceptionType>System.Data.SqlClient.SqlException</ExceptionType>

<StackTrace>
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)[...]
</StackTrace>

The Entity Framework "Invalid object name [class name]" exception means that the matching table for one of your classes hasn't been found.

In this case I'm trying to map the SomeClass to the underlying SQL table Map.Test:

[Table("Test", Schema="Map")]
public class SomeClass
{
    public int Id { get; set; }
}

The current version of EF7 (7.0.0-rc1-11953) does not have support for mapping using attributes in this way. Instead one must use Fluent configuration in the DbContext as follows:

public class MyContext : DbContext
{
    public DbSet<SomeClass> SomeClasses { get; set; }

    protected override void OnConfiguring(DbContextOptions options)
    {
        options.UseSqlServer(ConnectionString);
    }

    protected override void OnModelCreating(ModelBuilder modelBuilder)
    {
        base.OnModelCreating(modelBuilder);

        modelBuilder.Entity<SomeClass>().ForRelational().Table(tableName: "Test", schemaName: "Map");
    }
}

The mapping is configured fluently in the OnModelCreating method. For slightly more useful information about setting EF7 up see this link.

I hope this helps!

C# Async for Slugs

Eliot (Eliot Jones) — Sun, 07 Dec 2014 00:00:00 GMT

Fun fact for you dear reader, I've not been programming for long, my degree was in Chemistry and though I taught myself a little programming during my degree I was mainly focused on Chemistry. This means I've been programming professionally for less time than C# 5 has been released.

This is why I've found tutorials so far for the Async/Await features in C# 5 (Net 4.5) to be a little lacking, they generally assume the reader has been writing async programs prior to the release of the features.

As a solution to this I've endeavoured to write a guide to async for miserable slugs like myself who have not had exposure to async prior to these keywords.

Demo Scenario

We're going to stick with a very simple console app available on GitHub to make sure all the concepts can be understood by a slug.

Our Program.cs main method is shown below:

public static void Main(string[] args)
{   
    SlugService slugService = new SlugService();
    slugService.GetSlugs(generation: 1);
}

The GitHub version contains a lot more lines which allow the method calls to be timed with the built in stopwatch.

The Slug Service is tasked with retrieving the records of some slugs we've stored somewhere. Let's see what it does:

public IList<Slug> GetSlugs(int generation)
{
    Int64 generationalPrime = primeFinder.FindNthPrime(50000 + generation);

    IList<Slug> slugs = slugGetter.GetDataSlugs(generation);

    foreach (var slug in slugs)
    {
        slug.FavouritePrime = generationalPrime;
    }

    return slugs;
}

Again, the GitHub version contains more code related to timing methods with Stopwatch. This method does the following things:

Gets the favourite prime for this generation of slugs. (As we all know, slugs are huge prime number fans).
Get the slugs from our storage medium.
Set the prime for this generation of slugs.

The prime finder gets the nth prime for any n. It's written to be quite inefficient so we can replicate some external call or large calculation that blocks the program.

public Int64 FindNthPrime(int n)
{
    if (n < 1) throw new ArgumentException();

    List<int> primes = new List<int> { 2 };

    bool primeFound = false;
    int index = 3;

    while (!primeFound)
    {
        if (primes.Count >= n)
        {
            primeFound = true;
            break;
        }

        bool currentNumberIsPrime = true;
        foreach (int prime in primes)
        {
            currentNumberIsPrime = index % prime != 0;
            if (!currentNumberIsPrime) break;
        }

        if (currentNumberIsPrime) primes.Add(index);
        index += 2;
    }

    return primes[n-1];
}

If you don't understand the maths just ignore it, suffice to say when you call this method with an input of ~50,000 it takes around 8 seconds to return a result.

Finally we need to get our slugs from our data storage. I've not actually implemented a persistence medium for this demo so the SlugGetter does this instead:

public IList<Slug> GetDataSlugs(int generation)
{
    List<Slug> slugs = new List<Slug>();

    for (int i = 0; i < 1000; i++)
    {
        slugs.Add(new Slug
        {
            Id = i + 1,
            Name = "Slug" + i.ToString(),
            Generation = generation,
            CabbagesEaten = (i % 2 == 0) ? 1 : 0
        });
        Thread.Sleep(5);
    }
    return slugs;
}

The thread sleep in this method slows everything down so the program runs on a timescale which can be easily compared between runs.

The Problem with Synchronous Execution

Running the program with the Stopwatch related lines included gives this output:

The total run time is almost entirely due to the time taken to calculate the prime and the time taken to retrieve our slugs. The rest of the program takes about 2 milliseconds to run!

Unfortunately in the normal synchronous way of calling methods each method is run in turn causing our program to take a long time.

To visually illustrate the call stack I created this diagram:

By boosting the hardware of our computer we might achieve a more speedy slug retrieval (through a faster primefinder) however there's a limit to scaling up in this way. We need to move execution on to other threads so we're not being blocked by the PrimeFinder.

Parallel Execution != Async

This is the first conceptual hurdle for people new to C# wanting to learn about async and await. Since all examples of async use methods which return Task<T> it's easy to assume Task<T> is related to the async features.

Task<T> and related concepts in the System.Threading.Tasks namespace have actually been features of .NET since .NET 4.0. They form the core of the TPL (Task Parallel Library). We can therefore rewrite our SlugService with a parallel execution mode.

It would be nice if we could run the PrimeFinder on one of our cores in a worker task while the main execution ran the SlugGetter.

To create a Task for our PrimeFinder we can wrap our normal method in a new Task:

public Task<Int64> FindNthPrimeTask(int n)
{
    return Task.Run(() => FindNthPrime(n));
}

Task.Run() instantiates a new Task object and starts it running, it is a slightly nicer version of the .NET 4 TPL Task.Factory.StartNew() however you should use Task.Run().

Our method call now returns a Task so in our slug service we need to change the result of the PrimeFinder call to a Task<Int64> object:

public IList<Slug> GetSlugsParallel(int gen)
{
    Task<Int64> generationalPrimeTask = primeFinder.FindNthPrimeTask(50000 + gen);

    IList<Slug> slugs = slugGetter.GetDataSlugs(gen);

    Int64 generationalPrime = generationalPrimeTask.Result;
    foreach (var slug in slugs)
    {
        slug.FavouritePrime = generationalPrime;
    }
    return slugs;
}

Where we call Task.Result the main thread blocks execution while waiting for the results of the Task. This means we avoid a Race Condition where we use our prime before it has been properly calculated.

We will run the synchronous single threaded GetSlugs and then the GetSlugsParallel method:

As the results show, the run-time was almost entirely due to the time taken to calculate the prime (53 milliseconds was spent calling and returning the task). With the prime calculation taking place on another thread we were able to reduce run time.

It gets a bit harder to illustrate this call stack. First SlugService is called normally and performs a call to PrimeFinder as usual. However this returns in 50 milliseconds and the main thread goes on to run the SlugGetter which takes 6 seconds to return a result.

In the meantime a worker thread (in orange) runs the prime finder which takes about 9 seconds to run. This is running while the main thread is running SlugGetter. Therefore rather than adding runtimes together as we did for synchronous execution we take the highest time (this isn't always the case but works for this example).

Time for Async

So we've seen how we can use parallel execution without the async/await keywords. Let's introduce them to our program now.

For our PrimeFinder we do not need a new method, we should avoid creating asynchronous wrappers over synchronous CPU bound methods.

In SlugService we create a method with the async keyword in the method declaration. Async methods must return one of the following return types:

Task<T>
Task
void (don't return void from an async method due to exception handling)

This is achieved by changing the body of the method as follows:

public async Task<IList<Slug>> GetSlugsAsync(int gen)
{
    Int64 generationalPrime = await Task.Run(primeFinder.FindNthPrime(50000 + gen)).ConfigureAwait(false);

    IList<Slug> slugs = slugGetter.GetDataSlugs(gen);

    foreach (var slug in slugs)
    {
        slug.FavouritePrime = generationalPrime;
    }

    return slugs;
}

This is a bit confusing, the method is declared to return Task<IList<Slug>> but the method body returns an IList<Slug>. This is because the compiler can be thought of as wrapping the method up into a Task behind the scenes.

We have used await in the body of the method to tell the async method to wait for the Prime Finder to complete. Also note the use of ConfigureAwait(false) after the call to FindNthPrime(), this isn't strictly necessary in a Console App but will help to prevent deadlocks in UI/ASP.NET code by ensuring the completed task doesn't ask for the original thread back. We also call Task.Run() on the method rather than in the method implementation as recommended here.

Because Console Applications cannot have an async main method we have to 'unwrap' our async call at some point. To do this I created another method in SlugService that main actually calls to, Stephen Cleary outlines how to do this properly for console apps in the link but I'm using it here to illustrate a point:

public IList<Slug> GetSlugsAsyncCaller(int gen)
{
    var task = GetSlugsAsync(gen);
    Console.WriteLine("Returned from async");
    Thread.Sleep(1000);
    return task.Result;
}

This time the console output looks like this where async is run first, then synchronous and finally parallel execution:

As the results show, the async execution (14.5 seconds) takes just as long as the synchronous execution. But we're dealing with tasks, what gives?

Async works in a quite different way to parallel execution and async execution is not necessarily parallel. Let's look at a call stack diagram that attempts to explain this (again the blue arrows illustrate the main thread).

This is all terribly confusing. Basically at the point the code execution hits await in GetSlugsAsync it returns to the caller, in this case GetSlugsAsyncCaller, this is illustrated with the thicker blue arrow. The execution of that method (GetSlugsAsyncCaller) continues. It is then blocked by the call to task.Result and the rest of the GetSlugsAsync method executes in the normal way.

Let's reiterate that; await returns execution to the caller and the caller continues to execute normally. Because we started the PrimeFinder task this executes in parallel while we're back in the GetSlugsAsyncCaller method which is why the Thread.Sleep(1000) in that method doesn't increase the execution time. However the actual GetSlugsAsync method isn't parallel with respect to the flow inside its body, the await is blocking since the PrimeFinder task must return a result at this point.

Purpose of Async

So the async/await doesn't give us any (parallel) execution time benefit inside the GetSlugsAsync method itself. It's nice to continue to execute the calling method for a bit but this is of limited use. The calling method generally needs the result of the method called at some point in its body and because waiting for the result is blocking we don't gain much benefit.

So what's the point?

Well, the console app example falls down because it is not a good use of async. Async is like a virus that spreads through your code (not in a bad way), it only really works where the full call stack consists of async methods, you generally lose any benefit when you have to call .Wait() or .Result on an async method.

Because you can't have an async main method in your console app (it would return control to the Operating System which would mess everything up / close the app) it's hard to demonstrate the advantages in this type of app.

It's most obviously useful in WPF and ASP.NET applications. These both allow the methods called to return an async method at the root level to the caller:

In WPF the async method returns control to the UI loop on a call to await in the code-behind, this means the UI thread continues to be active while the async task is processing.
In ASP.NET the async controller action returns control to the application pool threads in IIS. This means the server can respond to other requests while the original request waits for its task to complete, generally using async I/O operations such as file I/O. This has a lot to do with I/O Completion Ports. Therefore if we only have 50 threads in our application pool to respond to website requests we're not tying them up dealing with long-running tasks and they can more quickly be used to respond to further requests. This does not mean the request is returned to the client early, just that the thread is returned to the server while the request processes.

The actual way in which true async requests are dealt with in WPF and ASP.NET is different to how it was illustrated in this tutorial. In this tutorial an extra thread was used and blocked while the prime was calculated because this was a CPU calculation and not a good candidate for async. True async operations such as file I/O don't use an extra thread.

See the section titled "What About the Thread Doing the Asynchronous Work?" here. To quote from that: "[...] no thread is required for true asynchronous work. No CPU time is necessary to actually push the bytes out ".

ASP NET MVC - Model Create With List of Model

Eliot (Eliot Jones) — Tue, 25 Nov 2014 00:00:00 GMT

This post runs through how to create a form for creating a model with a list of child models using MVC 5 and Razor.

I've been rather quiet on the blogging front lately, I've been working on getting my MVC 5 quiz application up and running on Azure, it's here if anyone wants to give it a go.

One thing I wanted to do while working on the site was allow for a model to be created along with a list of "sub" models; for example when creating a question it would be nice for a user to also create the answers on the same page.

Setup

To demo this we need a model which could "own" some collection of other items. I happened on the perfect example today in the Json.NET source:

From here

Yep, that's a rabbit with a pancake on its head.

Let's create our simple classes for this scenario:

namespace SandboxMvc.Pocos
{
    using System;
    using System.Collections.Generic;

    public class Rabbit
    {
        public int Id { get; set; }
        public string Name { get; set; }
        public int IrritationLevel { get; set; }
        public IList<Pancake> Pancakes { get; set; }
    }
    public class Pancake
    {
        public int Id { get; set; }
        public int Thickness { get; set; }
        public bool IsCrepe { get; set; }
    }
}

Controllers

We need two controller actions, first the create action get and post for the rabbit itself and then the partial view action for a pancake.

namespace SandboxMvc.Controllers
{
    using SandboxMvc.Pocos;
    using System.Collections.Generic;
    using System.Web.Mvc;
    public class RabbitController : Controller
    {
        [HttpGet]
        public ActionResult Create()
        {
            Rabbit rabbity = new Rabbit { Pancakes = new List<Pancake>() };
            return View(rabbity);
        }

        [HttpPost]
        public ActionResult Create(Rabbit model)
        {
            if (!ModelState.IsValid) return View(model);
            Session["Rabbit"] = model;
            return RedirectToAction(actionName: "Create");
        }
    }
}

I'm using the session for persistence here because I'm exceedingly lazy.

Views

Let's get Razor to scaffold our Create view first:

@model SandboxMvc.Pocos.Rabbit
@{int index = Model.Pancakes.Count;}
@using (Html.BeginForm())
{
    @Html.AntiForgeryToken()
    <div class="form-horizontal">
        @Html.ValidationSummary(true, "", new { @class = "text-danger" })
        <div class="form-group">
            @Html.LabelFor(model => model.Name, htmlAttributes: new { @class = "control-label col-md-2" })
            <div class="col-md-10">
                @Html.EditorFor(model => model.Name, new { htmlAttributes = new { @class = "form-control" } })
                @Html.ValidationMessageFor(model => model.Name, "", new { @class = "text-danger" })
            </div>
        </div>
        <div class="form-group">
            @Html.LabelFor(model => model.IrritationLevel, htmlAttributes: new { @class = "control-label col-md-2" })
            <div class="col-md-10">
                @Html.EditorFor(model => model.IrritationLevel, new { htmlAttributes = new { @class = "form-control" } })
                @Html.ValidationMessageFor(model => model.IrritationLevel, "", new { @class = "text-danger" })
            </div>
        </div>

        <!-- PANCAKES! -->
        <div id="pancake-group">
            <h4>Pancakes</h4>
        </div>
        @Ajax.ActionLink(linkText: "+", actionName: "_CreateFields", controllerName: "Pancake", routeValues: new { index = index }, ajaxOptions: new AjaxOptions
        {
            InsertionMode = InsertionMode.InsertAfter,
            UpdateTargetId = "pancake-group",
            OnSuccess = "indexIterate"
        },
        htmlAttributes: new { id = "addPancake", @class = "btn btn-default" })

        <div class="form-group">
            <div class="col-md-offset-2 col-md-10">
                <input type="submit" value="Create" class="btn btn-default" />
            </div>
        </div>
    </div>
}

The change we've made is where the comment excitably declares PANCAKES!.

We add a div <div id="pancake-group"> as an update target for our Ajax.

We then include the following Ajax.ActionLink:

@Ajax.ActionLink(linkText: "+", actionName: "_CreateFields", controllerName: "Pancake", 
routeValues: new { index = index }, 
ajaxOptions: new AjaxOptions
    {
        InsertionMode = InsertionMode.InsertAfter,
        UpdateTargetId = "pancake-group",
        OnSuccess = "indexIterate"
    },
    htmlAttributes: new { id = "addPancake", @class = "btn btn-default" })

This simply calls the _CreateFields action on our Pancake controller. It passes the index to add as a route value.

Of particular importance are the OnSuccess value of the AjaxOptions and the id of the htmlAttributes. These allow us to use the following Javascript to increment our link target:

var currentIndex = 0;

function indexIterate() {
    var newHref = $("#addPancake").attr("href");
    var newerHref = newHref.replace(/(?:index=)[0-9]+/i, "index=" + ++currentIndex);
    $("#addPancake").attr("href", newerHref);
};

Also notice the @index set near the top of the view code!

Pancake Controller

Now we add our pancake controller action and View:

public class PancakeController : Controller
{
    public ActionResult _CreateFields(Rabbit model, int? index)
    {
        ViewBag.Index = index ?? 0;
        return PartialView(model);
    }
}

This returns the partial view:

@model SandboxMvc.Pocos.Rabbit
@{
    int i = ViewBag.index ?? 0;
}

<div class="form-group">
    @Html.LabelFor(model => model.Pancakes[i].Thickness, htmlAttributes: new { @class = "control-label col-md-2" })
    <div class="col-md-10">
        @Html.EditorFor(model => model.Pancakes[i].Thickness, new { htmlAttributes = new { @class = "form-control" } })
        @Html.ValidationMessageFor(model => model.Pancakes[i].Thickness, "", new { @class = "text-danger" })
    </div>
</div>
<div class="form-group">
    @Html.LabelFor(model => model.Pancakes[i].IsCrepe, htmlAttributes: new { @class = "control-label col-md-2" })
    <div class="col-md-10">
        @Html.CheckBoxFor(model => model.Pancakes[i].IsCrepe, new { htmlAttributes = new { @class = "form-control" } })
        @Html.ValidationMessageFor(model => model.Pancakes[i].IsCrepe, "", new { @class = "text-danger" })
    </div>
</div>

Compile this and run it and you should be able to create all the pancakes you want on your rabbit.

Model Validation

When you have to use server side validation of your model, your sub items will disappear on validation if the validation fails.

Resolving this is luckily fairly simple. For your Create view for Rabbit add the following code inside your pancake-group div:

@for (int i = 0; i < index; i++)
{
    @Html.Action(actionName: "_CreateFields", controllerName: "Pancake", routeValues: new { index = i })
}

Conclusion

This tutorial showed how to add a list of models to the create for of another model and have that list submitted on post in addition to server side validation.

The full code for the files featured can be found as follows:

ASP.NET Identity 2.0 Tutorial - Password Reset and Roles

Eliot (Eliot Jones) — Mon, 13 Oct 2014 00:00:00 GMT

Given how much you all enjoyed the previous tutorial, i.e. not at all, I thought I'd write a follow up post. As promised this blog post extends the basic system we created in the main tutorial to add roles and a password reset function.

I won't show how to setup the emails for password reset because I'm too lazy to fill in the form for a free SendGrid account, however plugging emails in should be the easy bit.

Password Reset

The default template provides the controller actions and views for a full password reset function. The user can enter an email address at /Account/ForgotPassword. When posted the action below is called:

public async Task<ActionResult> ForgotPassword(ForgotPasswordViewModel model)
{
    if (ModelState.IsValid)
    {
        var user = await UserManager.FindByNameAsync(model.Email);
        if (user == null || !(await UserManager.IsEmailConfirmedAsync(user.Id.ToString())))
        {
            // Don't reveal that the user does not exist or is not confirmed
            return View("ForgotPasswordConfirmation");
        }
        string code = await UserManager.GeneratePasswordResetTokenAsync(user.Id.ToString());
        var callbackUrl = Url.Action("ResetPassword", "Account", new { userId = user.Id, code = code }, protocol: Request.Url.Scheme);      
        // await UserManager.SendEmailAsync(user.Id, "Reset Password", "Please reset your password by clicking <a href=\"" + callbackUrl + "\">here</a>");
        return RedirectToAction("ForgotPasswordConfirmation", "Account");
    }
    return View(model);
}

I have commented out the email sending part for the reasons mentioned above.

All we need to do is supply our UserManager with a token provider. From the code for the default UserManager:

/// <summary>
/// Used for generating reset password and confirmation tokens
/// </summary>
public IUserTokenProvider<TUser, TKey> UserTokenProvider { get; set; }

As usual Microsoft have made everything easy by giving us a nice interface to implement. This also means you can adapt your password reset to function however you like. For instance you could add an expiry date to the reset token or have a token which is a combination of a Guid and number or two Guids.

For my token provider I'm simply going to generate a new Guid to act as the reset token. This maps onto the nullable reset token field on my user table, I've hidden some stuff here because I don't want you stealing the super secure password hashes ("password" and "password1"):

Now we create a class which implements the required 4 methods for a token provider:

public class MyUserTokenProvider<TUser> : IUserTokenProvider<MyUser, string>  where TUser : class, IUser
{
    public Task<string> GenerateAsync(string purpose, UserManager<MyUser, string> manager, MyUser user)
    {
        Guid resetToken = Guid.NewGuid();
        user.PasswordResetToken = resetToken;
        manager.UpdateAsync(user);
        return Task.FromResult<string>(resetToken.ToString());
    }

    public Task<bool> IsValidProviderForUserAsync(UserManager<MyUser, string> manager, MyUser user)
    {
        if (manager == null) throw new ArgumentNullException();
        else {
            return Task.FromResult<bool>(manager.SupportsUserPassword);
        }
    }

    public Task NotifyAsync(string token, UserManager<MyUser, string> manager, MyUser user)
    {
        return Task.FromResult<int>(0);
    }

    public Task<bool> ValidateAsync(string purpose, string token, UserManager<MyUser, string> manager, MyUser user)
    {
        return Task.FromResult<bool>(user.PasswordResetToken.ToString() == token);
    }
}

The 0 value returned from NotifyAsync mirrors the way it's done in the TotpSecurityStampBasedTokenProvider included in the Identity library.
I don't call through to the store in the ValidateAsync method because the user argument will have been retrieved by the manager just before this method call so there's no point duplicating data access.
IsValidProviderForUserAsync should probably be implemented differently however this does what I want so why change? :)

The last thing we need to do is tell our UserManager about its swanky new token provider. You can either throw it in the overloaded constructor or Create method. Here I've put it in the constructor:

public class ApplicationUserManager : UserManager<MyUser>
{
    public ApplicationUserManager(IUserStore<MyUser> store)
        : base(store)
    {
        this.UserValidator = new MyUserValidator<MyUser, string>();
        this.UserTokenProvider = new MyUserTokenProvider<MyUser>();
    }
}

Now if you go to the link contained in the callbackUrl variable in the controller action you can reset a user's password.

Summary

Give your persistence layer some kind of a password reset token.
Create a class which implements IUserTokenProvider.
Pass that class to your UserManager class as the manager's UserTokenProvider.
?
Profit

Roles

The UserManager class has a set of boolean properties of the form:

/// <summary>
/// Returns true if the store is an IUserPasswordStore
/// </summary>
public virtual bool SupportsUserPassword
{
  get
  {
    return Store is IUserPasswordStore<TUser, TKey>;
  }
}

The booleans check which interfaces your UserStore implements. These booleans are used to throw descriptive exceptions where your store is lacking the methods the manager depends on. The full list is:

IUserTwoFactorStore
IUserPasswordStore
IUserSecurityStampStore
IUserRoleStore
IUserLoginStore
IUserEmailStore
IUserPhoneNumberStore
IUserClaimStore
IUserLockoutStore
IQueryableUserStore

The IUserStore has to be used by default so isn't included. Whenever we want to use a piece of the manager's functionality generally all we need to do is implement the missing interface on our store.

This extensibility comes in handy for the role store. With roles defined we can do things like this on our controllers:

[Authorize(Roles="Elevated Browser")]
public ActionResult Contact()
{
    return View();
}

This makes limiting access to certain actions very easy. Additionally because everything uses interfaces with generic parameters no assumptions are made about our persistence layer.

Let's say users gain points through some mechanism, when they get enough points they unlock certain privileges which can be represented by roles. Because we aren't tied to the Code First models for our store we can have this sort of model for our roles:

A Role requires a certain number of points and a user has points (which should probably be in a UserPoint table). This is not a brilliant way to implement this model but serves as a warning to future generations (example).

It would make sense to have a composite primary key of RoleId and UserId in the UserRole table however I've opted for a single column primary key in case UserRole evolves into some kind of business entity.

Again the fact I'm using Database-First Entity Framework shouldn't matter here because all that you need to worry about is how to implement your own version of the IUserRoleStore in your store.

Here's mine:

#region RoleStore
public Task AddToRoleAsync(MyUser user, string roleName)
{
    return Task.Factory.StartNew(() => ChangeRoleMembership(user, roleName, ChangeType.Add));
}

public Task<IList<string>> GetRolesAsync(MyUser user)
{
    using (var db = new EfDbContext())
    {
        User userDb = db.Users.Find(user.Id);

        IEnumerable<int> userRoleIds = userDb.UserRoles.Select(ur => ur.RoleId);

        IEnumerable<string> roleNames = db.Roles.Where(r => userRoleIds.Contains(r.Id)).Select(r => r.Name);

        return Task.FromResult<IList<string>>(roleNames.ToList());
    }
}

public Task<bool> IsInRoleAsync(MyUser user, string roleName)
{
    bool isInRole = false;
    using (var db = new EfDbContext())
    {
        Role roleDb = db.Roles.FirstOrDefault(r => string.Compare(r.Name, roleName) == 0);
        User userDb = db.Users.Find(user.Id);

        if (roleDb != null && userDb != null)
        {
            UserRole userRole = userDb.UserRoles.FirstOrDefault(ur => ur.RoleId == roleDb.Id);
            isInRole = userRole != null;
        }
    }

    return Task.FromResult<bool>(isInRole);
}

public Task RemoveFromRoleAsync(MyUser user, string roleName)
{
    return Task.Factory.StartNew(() => ChangeRoleMembership(user, roleName, ChangeType.Remove));
}
#endregion

The ChangeRoleMembership method and ChangeType enum are just things I've implemented and can be swapped out for your own methods.

Without any further changes you should be able to use the roles in your controller attributes.

Summary

Create some Roles feature in your persistence layer.
Add the IUserRoleStore interface methods to your user store.

On A Roll(e)

Hopefully this tutorial series shows how easy it is to make the Identity 2.0 system do what you want. If nothing else you should now have the information necessary to find out what the UserManager expects and provide classes that comply with its demands.

Roll Image from: http://nl.wikipedia.org/wiki/Bestand:Kaiserbroodjes1151.JPG. Yes I really did scroll through the roll category on Wikimedia... because I care.

ASP.NET Identity 2.0 Tutorial - EF Free (Post 2)

Eliot (Eliot Jones) — Sun, 12 Oct 2014 00:00:00 GMT

Note: This is the second post in a 2 post tutorial on implementing Identity 2.0 without using EF, for post 1 go here.

Now we have our User and UserStore classes we can change the UserManager and SignInManager our application uses. These classes are in the App_Start/IdentityConfig.cs file.

I split the classes out to their own files, ApplicationUserManager.cs and ApplicationSignInManager.cs respectively. In their unmodified state these classes inherit from the classes provided by the Identity library.

ApplicationUserManager

The manager provides many methods to use in our controllers, a few examples are:

public virtual Task<IdentityResult> ResetAccessFailedCountAsync(TKey userId);
public virtual Task<IdentityResult> ResetPasswordAsync(TKey userId, string token, string newPassword);
public virtual Task SendEmailAsync(TKey userId, string subject, string body);

The manager mainly delegates to classes it owns such as the user store to run these methods. The default template UserManager is shown below:

public class ApplicationUserManager : UserManager<ApplicationUser>
{
    public ApplicationUserManager(IUserStore<ApplicationUser> store)
        : base(store) { }

    public static ApplicationUserManager Create(IdentityFactoryOptions<ApplicationUserManager> options, 
        IOwinContext context) 
    {
        var manager = new ApplicationUserManager(
            new UserStore<ApplicationUser>(context.Get<ApplicationDbContext>()));

        // Configure validation logic for usernames
        manager.UserValidator = new UserValidator<ApplicationUser>(manager)
        {
            AllowOnlyAlphanumericUserNames = false,
            RequireUniqueEmail = true
        };

        // Configure validation logic for passwords
        manager.PasswordValidator = new PasswordValidator
        {
            RequiredLength = 6,
            RequireNonLetterOrDigit = true,
            RequireDigit = true,
            RequireLowercase = true,
            RequireUppercase = true,
        };

        [CONTENT REMOVED TO SAVE SPACE...]
        return manager;
    }
}

Because everything takes arguments based on interfaces we've already created our classes for, the rewrite is very simple. Firstly we replace all instances of ApplicationUser with our class which implements IUser, in our case MyUser:

public class ApplicationUserManager : UserManager<MyUser>
{
    public ApplicationUserManager(IUserStore<MyUser> store)
        : base(store)
    {
        this.UserValidator = new MyUserValidator<MyUser, string>();
    }

    public static ApplicationUserManager Create(IdentityFactoryOptions<ApplicationUserManager> options, 
        IOwinContext context)
    {
        var manager = new ApplicationUserManager(new MyUserStore<MyUser>());
        // Configure validation logic for usernames
        manager.UserValidator = new UserValidator<MyUser>(manager)
        {
            AllowOnlyAlphanumericUserNames = false,
            RequireUniqueEmail = true
        };

        // Configure validation logic for passwords
        manager.PasswordValidator = new PasswordValidator
        {
            RequiredLength = 3,
            RequireNonLetterOrDigit = false,
            RequireDigit = false,
            RequireLowercase = true,
            RequireUppercase = false
        };

        // Configure user lockout defaults
        manager.UserLockoutEnabledByDefault = false;

        manager.EmailService = new EmailService();
        manager.SmsService = new SmsService();

        return manager;
    }
}

I set the password requirements to be super weak for manual testing because typing secure passwords repeatedly was too much like hard work!

We also remove the default UserStore which depends on Entity Framework and insert our own MyUserStore (which in this tutorial also depends on EF but with no requirement to pass a DbContext to the constructor). Additionally all parts setting up 2 Factor Auth are removed.

An extra change in the Setup method is required, without it the call to UserManager CreateAsync(user, password) may fail when it tries to search by an email address which doesn't exist. It may not, it depends on how you implement interface methods in your store.

The CreateAsync method of UserManager calls the ValidateAsync method of the UserValidator object created by default (or explicitly in the Create method). If you decompile the validator you can see it calls the following methods of your store prior to actually creating the record:

FindByNameAsync
FindByEmailAsync

By default it expects you to return null from your store on these methods where the name or email doesn't exist. To avoid having the validator make assumptions about how you implement your methods you can supply your own class which implements IIdentityValidator as shown below:

public static ApplicationUserManager Create(
    IdentityFactoryOptions<ApplicationUserManager> options, 
    IOwinContext context)
{
    var manager = new ApplicationUserManager(new MyUserStore<MyUser>());
    manager.UserValidator = new MyUserValidator<MyUser, string>();
    ...
}

Where MyUserValidator : IIdentityValidator.

SignInManager

The sign in manager works with the user manager to validate user logins against the store and provide login information. The default sign in manager is pretty small which makes our change easy, shown below is the default implementation:

public class ApplicationSignInManager : SignInManager<ApplicationUser, string>
{
    public ApplicationSignInManager(ApplicationUserManager userManager, 
        IAuthenticationManager authenticationManager)
        : base(userManager, authenticationManager)
    {
    }

    public override Task<ClaimsIdentity> CreateUserIdentityAsync(ApplicationUser user)
    {
        return user.GenerateUserIdentityAsync((ApplicationUserManager)UserManager);
    }

    public static ApplicationSignInManager Create(IdentityFactoryOptions<ApplicationSignInManager> options, 
        IOwinContext context)
    {
        return new ApplicationSignInManager(context.GetUserManager<ApplicationUserManager>(), 
            context.Authentication);
    }
}

We can replace this with an almost identical replacement:

public class ApplicationSignInManager : SignInManager<MyUser, string>
{
    public ApplicationSignInManager(ApplicationUserManager userManager, 
        IAuthenticationManager authenticationManager)
        : base(userManager, authenticationManager) { }

    public static ApplicationSignInManager Create(
        IdentityFactoryOptions<ApplicationSignInManager> options, 
        IOwinContext context)
    {
        return new ApplicationSignInManager(context.GetUserManager<ApplicationUserManager>(), 
            context.Authentication);
    }
}

However here I removed the CreateUserIdentityAsync(ApplicationUser user) method.

I removed the method from the user also. This is because it seems odd to have the user take responsibility for creating itself with the manager. You're welcome to implement it if you wish but it just seemed strange to me.

Removing this breaks things in a couple of places. The most important one is in App_Start/Startup.Auth.cs:

// Enables the application to validate the security stamp when the user logs in.
// This is a security feature which is used when you change a password or add an external login to your account.  
OnValidateIdentity = SecurityStampValidator.OnValidateIdentity<ApplicationUserManager, ApplicationUser>(
    validateInterval: TimeSpan.FromMinutes(30),
    regenerateIdentity: (manager, user) => user.GenerateUserIdentityAsync(manager)
    )

Luckily this method just calls through to the CreateIdentityAsync method on the UserManager so we can change the above Startup code to:

OnValidateIdentity = SecurityStampValidator.OnValidateIdentity<ApplicationUserManager, MyUser>(
        validateInterval: TimeSpan.FromMinutes(30),
        regenerateIdentity: (m, u) => 
            m.CreateIdentityAsync(u, DefaultAuthenticationTypes.ApplicationCookie))

Rounding Up

We have implemented a bare-bones system, the code won't compile until you change all references to ApplicationUser to use your User implementation. Additionally many method calls will fail at run-time for things like 2 Factor and Claims-Based Authentication.

However if you remove enough code from the template controllers you should have a system which allows you to register and login/out. It also validates users based on the [Authorize] attribute on controllers.

I hope to write a post about implementing things like Password Reset and Roles but I've put this up now to help anyone who is stuck.

Conclusion

This tutorial showed how to use ASP.NET Identity 2.0 without Entity Framework. It does not use many of the new features of Identity 2.0 but shows how to create a basic login system with Identity 2.0.

The reliance of Identity 2.0 on EF Code First makes sense when viewed in the context of the retirement of Database/Model First in EF7. However it makes things difficult for people not using EF (Code First). There are many open source providers for other databases such as MongoDb available on NuGet and this tutorial hopefully explains a bit about how it all fits together.

In order to extend this simple identity system you'll need a good decompiler to see what's going on under-the-hood in the UserManager class when you run into problems. I can recommend dotPeek.

I hope this helps you, it aims to be the guide I wish I had when I started writing it. Any problems please leave a comment.

ASP.NET Identity 2.0 Tutorial - Entity Framework Free

Eliot (Eliot Jones) — Sat, 11 Oct 2014 00:00:00 GMT

So in this post I detailed how to use Asp Identity with Entity Framework Database First. However when I started a new MVC project and tried to follow the steps nothing worked. ASP Identity 2.0 has ruined everything by introducing many breaking changes.

This post will guide you on how to setup Identity 2.0 without Entity Framework. The original post was a lot longer but I've tried to cut down a lot of the rambling, for a great step-by-step guide to Identity 2.0 see this excellent CodeProject post by John Atten.

Create a New Project

Let's create a new MVC project using Individual User Accounts as shown below to see what the template gives us:

This gives us a familiar MVC project. At the root level is the partial class Startup.cs:

public partial class Startup
{
    public void Configuration(IAppBuilder app)
    {
        ConfigureAuth(app);
    }
}

This makes a call to the other part of this class Startup.Auth.cs defined in the App_Start folder. This does a lot of things including acting as an IoC container for our ApplicationDbContext , ApplicationUserManager and ApplicationSignInManager using Owin to create an instance for each web request. Additionally it sets up two-factor authentication for SMS verification of log-ins which is a new feature in Identity 2.0.

To remove the dependence on Entity Framework we need to implement everything without the Microsoft.AspNet.Identity.EntityFramework library.

A Case of Mistaken Identity

The ApplicationUserManager and ApplicationSignInManager mentioned are in the file IdentityConfig.cs. This file contains the following 4 classes:

EmailService
SmsService
ApplicationUserManager
ApplicationSignInManager

For now we can ignore the EmailService and SmsService. This is where you'd integrate something like SendGrid or your own SMTP server to send emails.

Back to Basics

The layout of classes and what relies on what is actually fairly simple, it just takes a bit of thinking about. This image outlines what we're talking about:

We have 4 main classes to worry about:

User - it's up to us to implement this using the IUser interface. The EF version uses the IdentityUser class but we can't use this.
UserStore - the user store handles saving and retrieving data from our storage medium. This could be anything you can think of. In this tutorial I'm using EF Database First but it won't affect the EF-free nature of this tutorial.
UserManager - this is implemented for us by the Microsoft.AspNet.Identity.Core library, it's up to us to provide it with a suitable user and user store implementation.
SignInManager - this is implemented for us by the Identity.Owin library, we must provide it a suitable user manager and user implementation.

User

So Identity 2.0 deals with a user implementation. In the new MVC project template it uses the IdentityUser class. This is in the IdentityModels.cs file. This file contains 2 classes:

ApplicationUser
ApplicationDbContext

Any hope we have of easily separating our project from a dependency on EntityFramework is dashed here because IdentityUser is an EF specific class:

public class ApplicationUser : IdentityUser
{
    public async Task<ClaimsIdentity> GenerateUserIdentityAsync(UserManager<ApplicationUser> manager)
    {
        var userIdentity = 
            await manager.CreateIdentityAsync(this, DefaultAuthenticationTypes.ApplicationCookie);
        return userIdentity;
    }
}

By moving everything back onto Core we only have to implement an IUser<TKey> class where TKey is the type of the primary key (e.g. int, Guid, etc).

I implemented the class MyUser as follows:

namespace TutorialIdentity.Identity
{
    using Microsoft.AspNet.Identity;
    using System;
    public class MyUser : IUser<int>, IUser
    {
        public int Id { get; set; }
        string IUser<string>.Id{ get { return Id.ToString(); } }

        public string UserName { get; set; }
        public string Email { get; set; }

        public string PasswordHash { get; set; }
        public string SecurityStamp { get; set; }
        public Guid? PasswordResetToken { get; set; }
    }
}

This class is then mapped onto an underlying class in the EF Db First model but could be stored in anything.

I use both the IUser<int> and IUser (equivalent to IUser<string>) interface in order to make life easier when integrating with the existing classes and interfaces.

UserStore

The user store is required to implement several interfaces. Firstly it must implement the IUserStore interface. From the code for the UserManager class:

/// <summary>
/// Persistence abstraction that the UserManager operates against
/// 
/// </summary>
protected internal IUserStore<TUser, TKey> Store { get; set; }

The UserManager takes the store in its constructor.

I created a class MyUserStore as follows:

public class MyUserStore<TUser> : IUserStore<MyUser> where TUser : IUser
{
    ...
}

IUserStore

In MyUserStore this interface is implemented as follows:

public Task CreateAsync(MyUser user)
{
    User userDb = ToUser(user);
    return Task.Factory.StartNew(() =>
    {
        SaveUser(userDb);
    });
}

public Task DeleteAsync(MyUser user)
{
    throw new System.NotImplementedException();
}

public Task<MyUser> FindByIdAsync(string userId)
{
    return Task.Factory.StartNew<MyUser>(() => FindUser(userId, FindKeyType.Id));
}

public Task<MyUser> FindByNameAsync(string userName)
{
    return Task.Factory.StartNew<MyUser>(() => FindUser(userName, FindKeyType.Name));
}

public Task UpdateAsync(MyUser user)
{
    return Task.Factory.StartNew(() =>
    {
        using (var db = new EfDbContext())
        {
            User userDbOrig = db.Users.Find(user.Id);
            userDbOrig.UserName = user.UserName;
            userDbOrig.PasswordHash = user.PasswordHash;
            userDbOrig.Email = user.Email;
            db.Entry(userDbOrig).State = EntityState.Modified;
            db.SaveChanges();
        }
    });
}

I haven't finished writing UpdateAsync and DeleteAsync methods since this is just a throwaway tutorial store. These methods call other methods which use the EF DbContext. However they could just as easily call a webservice or MongoDb or a text file storage mechanism (which would be really weird but I give the example to show that the world is your oyster).

In its current state our UserStore will be accepted by the UserManager at compile time however we can do nothing useful with it at runtime. To get to the point where logins are working we need some more interfaces on our store.

The full list to get to the point where logging in / registering works is:

IUserPasswordStore<MyUser>
IUserEmailStore<MyUser>
IUserLockoutStore
IUserTwoFactorStore
IUserSecurityStampStore<MyUser>

Note that I just chucked in TKey of string where it was required. When you're writing your store you should take time to make sure your TKey is being passed around properly. All interfaces in the Identity library have IInterface<TUser> : IInterface<TUser, string> and IInterface<TUser, TKey> versions.

IUserPasswordStore

This requires you to implement the following:

#region PasswordStore
public Task<string> GetPasswordHashAsync(MyUser user)
{
    return Task.Factory.StartNew<string>(() =>
    {
        using (var db = new EfDbContext())
        {
            return FindUser(user.Id.ToString(), FindKeyType.Id).PasswordHash;
        }
    });
}

public Task<bool> HasPasswordAsync(MyUser user)
{
    return Task.Factory.StartNew<bool>(() =>
    {
        using (var db = new EfDbContext())
        {
            return !string.IsNullOrEmpty(db.Users.Find(user.Id).PasswordHash);
        }
    });
}

public Task SetPasswordHashAsync(MyUser user, string passwordHash)
{
    return Task.Factory.StartNew(() =>
    {
        user.PasswordHash = passwordHash;
    });
}
#endregion

Again these are awful store methods just written to get something working. Don't Use These!

IUserEmailStore

Implementing the following methods:

#region EmailStore
public Task<MyUser> FindByEmailAsync(string email)
{
    return Task<MyUser>.Factory.StartNew(() =>
    {
        using (var db = new EfDbContext())
        {
            User userDb = db.Users.First(u => u.Email == email);
            return ToMyUser(userDb);
        }
    });
}

public Task<string> GetEmailAsync(MyUser user)
{
    return Task<string>.Factory.StartNew(() => return user.Email);
}

public Task<bool> GetEmailConfirmedAsync(MyUser user)
{
    return Task<bool>.Factory.StartNew(() => return true);
}

public Task SetEmailAsync(MyUser user, string email)
{
    return Task.Factory.StartNew(() => user.Email = email);
}

public Task SetEmailConfirmedAsync(MyUser user, bool confirmed)
{
    throw new NotImplementedException();
}
#endregion

This doesn't implement email confirmation properly but this is where you'd do so if your site is using it.

I won't show full implementations for all the interfaces here, because the way they're implemented is rubbish and you can get Visual Studio to tell you what you should be implementing (anyone could write better methods than these!).

The full code for my (really, really bad) user store is here. As you can see most of it is not implemented yet since a lot of these methods are not used if you remove 2 factor / claims-based auth.

The following 2 interfaces:

IUserLockoutStore<MyUser, string>,
IUserTwoFactorStore<MyUser, string>

Must be implemented even if you're not using lockouts (you should) or 2 factor auth. You can just use the method returning bool to tell the user/sign-in manager they're not enabled, like so:

public Task<bool> GetTwoFactorEnabledAsync(MyUser user)
{
    return Task.Factory.StartNew<bool>(() => false);
}

This post couldn't grow much longer and still be readable so the next two classes are discussed in the next post. Once you've implemented all the interfaces on your store to your satisfaction continue reading.

Minify Json using JSON.NET

Eliot (Eliot Jones) — Thu, 25 Sep 2014 00:00:00 GMT

In my current project I'm using Json stored in the SQL Server database to replicate the storage capability of the browser Session but allowing the data to persist on the user reloading/leaving the page.

One of the challenges is to store the serialized object in as few characters as possible while preserving meaning.

Without touching the settings, Json.Net stores the strings without extraneous whitespace, but even serializing a fairly small aggregate object can use a lot of characters.This post details a couple of the features I've used to minimise the number of characters.

Store Booleans as Integers in Json

JSON.NET supports custom conversions using a class which inherits from JsonConverter:

public class BooleanConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return typeof(bool).IsAssignableFrom(objectType);
    }

    public override void WriteJson(JsonWriter writer,
        object value, JsonSerializer serializer)
    {
        bool? source = value as bool?;
        if (source == null) { return; }

        int valueAsInt = ((bool)source) ? 1 : 0;

        writer.WriteValue(valueAsInt);
    }

    public override object ReadJson(JsonReader reader, Type objectType,
        object existingValue, JsonSerializer serializer)
    {
        bool returnValue = (Convert.ToInt32(reader.Value) == 1);

        return returnValue;
    }
}

This converter will store booleans as 1 and 0 rather than true and false; this saves 3 and 4 characters respectively.

Shorten Field and Property Names

By using the built in [JsonProperty("Name")] we can save space in the serialized object. If we define one letter aliases for our properties we save a lot of space, especially when serializing an array of objects:

[JsonProperty("b")]
public bool IsAThing { get; set; }

If the full object has IsAThing = false the string representing the serialized object stores this as "b":0 which is the shortest representation possible (unless you set the default value to false and only serialize where the boolean value is true).

Serializing an HtmlString

This isn't a space saving tip but where you need HtmlStrings to be serialized and deserialized you can use the converter below:

public class HtmlStringConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return typeof(IHtmlString).IsAssignableFrom(objectType);
    }

    public override void WriteJson(JsonWriter writer, 
        object value, JsonSerializer serializer)
    {
        IHtmlString source = value as IHtmlString;
        if (source == null) { return; }            
        writer.WriteValue(source.ToString());
    }

    public override object ReadJson(JsonReader reader, Type objectType, 
        object existingValue, JsonSerializer serializer)
    {
        return new HtmlString((string)reader.Value);
    }
}

With thanks to this StackOverflow answer.

Simple Introduction to MVC Ajax (Post 3)

Eliot (Eliot Jones) — Tue, 16 Sep 2014 00:00:00 GMT

Now we have partial views loading forms using Ajax and we can call controller methods using Ajax it's time to put everything together.

We're going to start with a blank scenario to make it easier to follow. This makes this a very long post but it puts together everything we learned.

Code files

I managed to coerce PHP into creating a new route on my website. You can see the full code for the classes/views in this demo as follows:

Setting up the test environment

Cats are spiteful creatures, full of barely concealed malice. It's one of their most loveable qualities, let's make a view to show the details of our cat and the grudges it holds.

Let's add a class to represent our cat (Pocos/Cat.cs):

public class Cat
{
    public int Id { get; set; }
    public string Name { get; set; }
    public int AngerLevel { get; set; }
    public DateTime BirthDate { get; set; }

    public List<Grudge> Grudges { get; set; }
}

Where the Grudge class is represented by the following (Pocos/Grudge.cs):

public class Grudge
{
    public int Id { get; set; }
    public string TargetName { get; set; }
    public string Reason { get; set; }
}

To view the details of our cat we need a controller. In this demo I'm using the Session as our backing data store. This is terrible and should never be done, however this isn't about how to data bind. Add the controller (Controllers/CatController.cs):

public ActionResult Details()
{
    Cat cat = new Cat
    {
        Id = 1,
        BirthDate = new DateTime(2005, 10, 3),
        Name = "Mr. Tibbles",
        AngerLevel = 100,
        Grudges = new List<Grudge> { 
            new Grudge 
            { 
                Id = 25, 
                TargetName = "All Humans" , 
                Reason =  "Opposable thumbs" 
            } 
        }
    };
    Session["Cat"] = cat;

    return View(cat);
}

Finally a view for our cat, unfortunately the syntax highlighter mangles this code. We won't worry about showing Grudges here (Views/Cat/Details.cshtml):

@model SandboxMvc.Pocos.Cat
<h2>Cat</h2><div>
    <dl class="dl-horizontal">
        <dt>@Html.DisplayNameFor(model => model.Name)</dt>
        <dd>@Html.DisplayFor(model => model.Name)</dd>
        <dt>@Html.DisplayNameFor(model => model.AngerLevel)</dt>
        <dd>@Html.DisplayFor(model => model.AngerLevel)</dd>
        <dt>@Html.DisplayNameFor(model => model.BirthDate)</dt>
        <dd>@Html.DisplayFor(model => model.BirthDate)</dd>
</dl></div>

The Book of Grudges

We need a partial view to display our cat's grudges.

We're going to include an Html.Action in our view so we need a method in the Controller (Controllers/GrudgeController.cs) to call:

public PartialViewResult _GrudgesForCat(int id)
{
    List<Grudge> grudges = ((Cat)Session["Cat"]).Grudges;
    return PartialView(grudges);
}

We also need to include the Partial View in our Details view:

<div id="BookOfGrudges">
    @Html.Action(actionName: "_GrudgesForCat", 
    controllerName: "Grudge", 
    routeValues: new { id = Model.Id })
</div>

And finally we need a PartialView for this action (Views/Grudge/_GrudgesForCat.cshtml):

@model List<SandboxMvc.Pocos.Grudge>
@for (int i = 0; i < Model.Count; i++)
{
    @Html.DisplayFor(m => m[i].TargetName) <br/>
    @Html.DisplayFor(m => m[i].Reason) <br/><br/>
}

When you run this you should be able to see a representation of the Cat we created, with associated Grudge(s).

The list grows

It would be nice to create new grudges (if you're a cat). We'd like to do that without leaving our page.

We're going to include a partial view of a form to create new grudges. Modify the Details view to add the following:

<div id="AddToGrudgeList">
    @Html.Action(actionName: "_Create",
    controllerName: "Grudge",
    routeValues: new { id = Model.Id });
</div>

We then need the backing Action on our Grudge controller for both Get and Post:

public PartialViewResult _Create (int id) 
{
    return PartialView();
}

[HttpPost]
public ActionResult _Create(Grudge model)
{
    ((Cat)Session["Cat"]).Grudges.Add(model);
    return RedirectToAction(actionName: "_GrudgesForCat", routeValues: new { id = 9 });
}

The create view is a fairly long form but look a bit like this. It's important that you specify the action name and controller (Views/Grudge/_Create.cshtml):

@model SandboxMvc.Pocos.Grudge
@using (Html.BeginForm(actionName: "_Create", controllerName: "Grudge"))
{
    @Html.AntiForgeryToken()
    <div class="form-horizontal">
        <h4>Grudge</h4>
        @Html.ValidationSummary(true, "", new { @class = "text-danger" })
        <div class="form-group">
            [FORM INPUTS]
        </div>

        <input type="submit" value="Create" class="btn btn-default" />
    </div>
}

When we click to create a grudge the action "_Create" is called and the Grudge is added to the list. However the result opens in a new page rather than updating our Book of Grudges.

Make it more... Ajax

Everything we've seen so far has been normal MVC using HTML only. All actions are called on the server by performing a full post/get.

Make sure you have Ajax installed in your project (see post 1).

In order to dynamically update our Book of Grudges using Ajax we need to make one change, change the @Html.BeginForm to @Ajax.BeginForm (Views/Grudges/_Create.cshtml):

@model SandboxMvc.Pocos.Grudge
@using (Ajax.BeginForm(actionName: "_Create", 
    controllerName: "Grudge", 
    ajaxOptions: new AjaxOptions { UpdateTargetId = "BookOfGrudges" }))
{
    @Html.AntiForgeryToken()
    <div class="form-horizontal">
        [FORM]  
        <input type="submit" value="Create" class="btn btn-default" />
    </div>
}

That's it, now your Book of Grudges is updated using partial page updates.

Conclusion

Over the past 3 posts I've walked through how to get started with Ajax in MVC 5. This series addresses some of the main pain points involved in getting this working with Partial Views and forms.

The Ajax included in MVC makes it possible to have a fully featured Ajax application without having to touch a line of Javascript, a positive because js is a language only people with a true passion for ducks could love.

If you have any questions or tips please add a comment; I'd be interested to hear the many, many things I got wrong in this series.

The solution view of the finished project is shown here:

Cat picture from Wikipedia.

Simple Introduction to MVC Ajax (ASP.NET MVC 5)

Eliot (Eliot Jones) — Sun, 14 Sep 2014 00:00:00 GMT

The jQuery Ajax built into ASP.NET MVC is very powerful but it's hard to know exactly where to start. This tutorial is a result of a few hours investigation so it's written from the point of view of a beginner.

I intend to upload code samples at some point but I need to add a code hosting feature to my website.

Setting up our test environment

We're going to use a very simple example to work with only the details we need. As always we're using a dog class, here's a picture of a dog:

The first thing we need to do is set our project up with Ajax. Right click your project and open the Nuget Package Manager. Search for "Ajax" and install the Microsoft jQuery Unobtrusive Ajax package.

This adds the jquery.unobtrusive-ajax.min.js file to our Scripts folder. We need to include this in our webpage. The easiest way to do this is to add it to our BundleConfig class (in the App_Start folder):

bundles.Add(new ScriptBundle("~/bundles/jquery").Include(
           "~/Scripts/jquery-{version}.js", 
           "~/Scripts/CustomScripts/StickyFooter.js",
           "~/Scripts/jquery.unobtrusive-ajax.min.js"));

This should be included in your master _Layout page using @Scripts.Render("~/bundles/jquery").

Creating a test page

With the following dog class:

public class Dog
{
    public int Id { get; set; }
    public string Name { get; set; }
    public DateTime BirthDate { get; set; }
}

And the following controller method (DogController.cs):

public ActionResult Details()
{
    Dog dog = new Dog{Id = 7, Name = "Lord woofington",
        BirthDate = new DateTime(year: 2005, month: 06, day: 27)};
    return View(dog);
}

We can quickly get a view to work with. Before we can navigate to the action we need a view. This is the view the Wizard created for the Dog class in Views/Dog/Details.cshtml:

@model JunkCode.Dog
<h2>Dog Details</h2>
<div>
    <dl class="dl-horizontal">
        <dt>@Html.DisplayNameFor(model => model.Name)</dt>
        <dd>@Html.DisplayFor(model => model.Name)</dd>
        <dt>@Html.DisplayNameFor(model => model.BirthDate)</dt>
        <dd>@Html.DisplayFor(model => model.BirthDate)</dd>
    </dl>
</div>

Our first simple Ajax call

The simplest thing we can do is call a controller method using an ActionLink. We add the following code to our view:

@Ajax.ActionLink(linkText: "Call An Action", 
                    actionName: "Bark", 
                    ajaxOptions: new AjaxOptions { UpdateTargetId = "ajaxtarget", 
                                                    InsertionMode = InsertionMode.InsertAfter })

<div id="ajaxtarget"></div>

Some points:

Named parameters using [text]: have been a feature of Visual Studio since 2010 and are a nifty little feature to make your code more readable.
This ActionLink presents a link with the given linkText and when clicked calls the Action (given by actionName) on the Dog Controller.
The AjaxOptions allow you to modify the exact behaviour of the Ajax call.

We then need an Action on our Controller for our link to actually call:

public string Bark(){ 
    return "Woof!"; 
}

When we click this the text "Woof!" is added to our ajaxtarget div. Clicking repeatedly appends the text to our div.

This is pretty nifty, but what else can we do?

Go to the Next Post to find out about using PartialViews and Ajax Forms.

Simple Introduction to MVC Ajax (Post 2)

Eliot (Eliot Jones) — Sun, 14 Sep 2014 00:00:00 GMT

So far we have seen how to add a very simple ActionLink and attach actions. Now let's load a PartialView using Ajax. Also we want to load a partial view with a different model into our page.

Getting a Partial View using ActionLink

We're going to deal with an object a Dog could "own", so we create a collar class. Let's accessorize our dog with some fabulous collars / add an ActionLink to the Dog/Details view (Details.cshtml):

@Ajax.ActionLink(linkText: "Call An Action Returning a partial view",
    actionName: "GetCollarCreate",
    routeValues: new { id = @Model.Id },
    ajaxOptions: new AjaxOptions
    {
        UpdateTargetId = "ajaxpartialtarget",
        InsertionMode = InsertionMode.Replace
    })
<div id="ajaxpartialtarget"></div>

We can pass the Id of the dog in using RouteValues as normal. The InsertionMode.Replace is default so doesn't need to be defined explicitly, I'm just doing it because it doesn't hurt to be reminded. This ActionLink calls the following method on the Dog Controller (DogController.cs):

public ActionResult GetCollarCreate(int id){
    return RedirectToAction(actionName: "_Create", 
        controllerName: "Collar", 
        routeValues: new { id });
}

This then directs us to an action on our Collar Controller class (CollarController.cs):

[HttpGet]
public PartialViewResult _Create(int id){
    Collar collar = new Collar { DogId = id };
    return PartialView(viewName: "_Create", model: collar);
}

Where our collar is a simple view model:

public class Collar
{
    public int DogId { get; set; }

    public string Color { get; set; }

    public bool HasTag { get; set; }
}

We will create a more detailed view in the next part but for now we have this simple view (Create.cshtml):

@model JunkCode.Collar
<div> Loaded the partial view for dog with Id = @Model.DogId </div>

When we build and run this we get the result we want, the view is loaded using an Ajax request, the dog woofs for joy:

Hiding the PartialView again

It would be nice to remove the partial view if we change our mind about whatever Action we're calling, I added this link in our Collar/Create view (Create.cshtml):

@Ajax.ActionLink(linkText: "Hide",
    actionName: "_HideCreate",
    ajaxOptions: new AjaxOptions
    { UpdateTargetId = "ajaxpartialtarget" })

This calls the Controller method (CollarControler.cs):

[HttpGet]
public void _HideCreate() { ; }

The coding of the UpdateTargetId into another view is kinda crappy, but I'm sure you can think of a way to improve it if it makes you incandescent with rage.

Loading a form in our PartialView

We modify the "GetCollarCreate" method in our Dog controller to return the results of our new Collar controller method (DogController.cs):

return RedirectToAction(actionName: "_Create2",
        controllerName: "Collar",
        routeValues: new { id });

The _Create2 method returns a PartialView (CollarController.cs):

[HttpGet]
public PartialViewResult _Create2(int id)
{
    Collar collar = new Collar { DogId = id };
    return PartialView(viewName: "_Create2", model: collar);
}

The Create2 view was created by the Scaffolding tool but looks a bit like this (Create2.cshtml):

@model Questions.JunkCode.Collar
@using(Html.BeginForm("_Create2", "Collar", new {}))
{
    @Html.AntiForgeryToken()

    <div class="form-horizontal">
        @Html.ValidationSummary(true)
        <div class="form-group">
            [MORE STUFF HERE...]
        </div>

        <div class="form-group">
        <div class="col-md-offset-2 col-md-10">
            <input type="submit" value="Create" class="btn btn-default" />
        </div>

    </div>
    </div>
}
<div id="statusMessage"></div>

We can then define the _Create2 post method in our Collar Controller (CollarController.cs):

[HttpPost]
public string _Create2(int id, Collar model)
{
    // Call some repository method to create the collar
    return "Collar Created!";
}

When we load this controller we get the following result:

So far we've used an Ajax ActionLink and loaded an HTML form into our page using Ajax, how about getting the form to submit using Ajax? This is the tricky bit to get right and is the topic of the next post

Error Creating a DACPAC in Visual Studio 2013 from SQL Server 2014

Eliot (Eliot Jones) — Sun, 31 Aug 2014 00:00:00 GMT

Working on projects in your spare time is great because you're free to pursue every little diversion; it means you make no real progress with your actual project but you learn a lot.

When I had to add a column to one of my database tables I decided to investigate creating a DACPAC(1). A DACPAC allows you to turn your existing database into a database project in Visual Studio which means your schema can be placed under source control. Changes can then be pushed to your database in a friendly wizard format.

To create a DACPAC you right click on your existing database and choose Extract Data Tier Application under tasks:

The wizard then runs through the steps to create the DACPAC.

Next create a new SQL Server Database project in your Visual Studio solution:

Then right click the project and choose Import -> Data-Tier Application. Choose the dacpac you extracted from SQL Server and then click Start.

If everything goes according to plan you should successfully import the dacpac. If this is the case, use this post to learn what you can do with your dacpac.

However I got the following error:

Internal Error. The database platform service with type Microsoft.Data.Tools.
Schema.Sql.Sql120DatabaseSchemaProvider is not valid. You must make sure the
service is loaded, or you must provide the full type name of a valid database
platform service.

I'm using Visual Studio 2013 Express with SQL Server 2014 Express (I'm poor :) ), after a bit of searching I found the problem was an outdated version of the Data Tools. To fix this do the following, click Tools -> Extensions and Updates:

Then choose Updates from the window. You should see an update with a name similar to SQL Server Data Tools in Product Updates (I've already updated mine so it's not shown):

Hopefully this fixes your issue.

Thanks to Steve Wade for telling me about dacpacs.

ASP.NET Identity 1.0 - Database First

Eliot (Eliot Jones) — Sat, 16 Aug 2014 00:00:00 GMT

Note: This tutorial applies to Identity 1.0 which is the previous version of ASP.NET Identity. I have a newer post on the same subject here

If you've read my previous posts you probably know I'm working on some EF Database first stuff, this seems to be the least popular way to use EF but to me seems like a common business use-case for it.

The MVC template in Visual Studio comes with the new Asp.NET Identity system built in, but it's designed to be used with Code First. This leads to people asking how to use Identity with Database First? The suggested soultion is to use the table structure provided when you use Code First and port it to your pre-existing database.

This seemed like an unpleasant approach to me (and a lot of work), especially because I think the database layout for those tables is kinda sucky. This is when it occurred to me that we're basically trying to implement Identity without using what the documentation thinks of as Entity Framework.

When you Google how to do that, you get this great post from Mark Johnson. You can follow his approach and just substitute his use of Dapper for use of Entity Framework. for instance:

public Task CreateAsync(User user)
{
    if (user == null)
    {
        throw new ArgumentNullException("user was null");
    }

    return Task.Factory.StartNew(() =>
        {
            db.Users.Add(user);
            db.SaveChanges();
        });
}

public Task DeleteAsync(User user)
{
    User checkUser = db.Users.Find(user.Id);

    if (user == null || checkUser == null)
    {
        throw new ArgumentNullException("user was null");
    }

    return Task.Factory.StartNew(() =>
    {
        db.Users.Remove(user);
        db.SaveChanges();
    });
}

Where db is your DbContext. Hopefully this helps you as much as it helped me, it's a far nicer solution than rolling your own Auth system.

AOB (1)

Eliot (Eliot Jones) — Sun, 10 Aug 2014 00:00:00 GMT

AOB here is used to mean 'any other business' rather than the Swedish pop group Ace of Base or the Accessory olfactory bulb.

This a blog post to summarise a few things which don't make a whole blog post in their own right.

MarkdownPad 2

For those of you who don't know, markdown is readable HTML syntax so you can write properly structured HTML in a human friendly way.

The conversion to HTML is then performed by the markdown library in your language of choice using a mega-regex.

The format is used by sites like StackOverflow and GitHub for comments/text and I also use it for my blog posts.

So after Scott Hanselmann mentioned it in his blog I downloaded MarkdownPad 2.

Beforehand I was using Notepad++ to write Markdown and it was horribly long winded due to the poor choice of application. Now writing Markdown is about as easy as using Word (so go download it if you use markdown).

Ceci n'est pas une pipe (trans. C# Equality is Weird)

I was trying to write a method to find where properties had been set back to their default value on an object. Reflection seemed like an obvious choice since I wanted a one-size fits all method.

I ran into a very odd problem where C# was convinced that 0 == 0 is false, or so I thought. The problem was based on my lack of understanding of how the equality operator actually works, but provided a good learning experience.

The actual comparison that troubled me could be simplified to:

int num = 0;
object num1 = num;
object num2 = num;
num1 == num2 //false

See this StackOverflow question for the answer. Basically for the object type, == checks that both objects occupy the same location in memory (ReferenceEquals).

For the example above the integer is boxed (converted to the reference type object) in two different locations so == returns false since they occupy different locations in memory.

If Equals(num1, num2) is used instead, the method returns true.

This lead to the even more interesting problem posed by Eric Lippert in the comment on the question. Why does the following occur (this is output from Immediate Window in Visual Studio)?

short myshort = 0;
0
int myint = 0;
0

myshort.Equals(myint)
false
myint.Equals(myshort)
true
myint == myshort
true

For the full explanation see his post on Coverity which explains what's going on in detail. There's also this post by Jon Skeet for further reading.

ActionLink MVC

Just for future reference here's an ActionLink from MVC4:

@Html.ActionLink("Edit", "Edit", "ControllerName", 
    routeValues: new { first="duck", @Model.Id, other = "rgs", thing = @Model.Id}, 
    htmlAttributes: new { @class = "cssclass" })

The first parameter is link text, second is Action Method and third is controller name.

Route values are added as an anonymous type. They are compared against the defaults in RouteConfig.cs and any that do not match the pattern will be added in the querystring. For example:

routes.MapRoute(
                name: "Default",
                url: "{controller}/{action}/{id}",
                defaults: new { controller = "Home", action = "Index", id = UrlParameter.Optional }
            );

Will map 'other', 'first' and 'thing' into the querystring, whereas the unnamed value will be the id.

{more url here}/Edit/-2147483647?first=duck&other=rgs&thing=-2147483647

Sea Kayaking

Just in case concerned readers were thinking I spent my entire time coding, I've uploaded a couple of pictures from my recent kayaking trip to Old Harry Rocks. I've got a sit on-top kayak from GoSea which is great for exploring the coast but heavy-going once the wind picks up.

In total the distance was about 7km there and back, you can see Old Harry in the background here.

I've also recently taken it to Lulworth and paddled along to Durdle Door which was great and highly recommended if you get a chance.

Metadata For Entity Framework MVC - Database First

Eliot (Eliot Jones) — Sat, 09 Aug 2014 00:00:00 GMT

The model validation in MVC is brilliant but for best usage requires Attributes to be added to your model.

If you use Entity Framework to generate your model you are given classes which will be overwritten when you regenerate the model as seen in the warning automatically appended to your file:

//------------------------------------------------------------------------------
// <auto-generated>
// This code was generated from a template.
//
// Manual changes to this file may cause unexpected behavior in your application.
// Manual changes to this file will be overwritten if the code is regenerated.
// </auto-generated>
//------------------------------------------------------------------------------

The way around this is to use a Metadata Class. Because an Entity Framework model class is defined as a partial class you can extend it.

For the following model class:

namespace Project.EntityFramework
{
    using System;
    using System.Collections.Generic;

    public partial class Dog
    {
        public Dog()
        {
            this.Collars = new HashSet<Collar>();
        }

        public int Id { get; set; }
        public string Name { get; set; }

        public virtual ICollection<Collar> Collars { get; set; }
    }
}

You can extend the partial class as long as your class is in the same namespace. The specific metadata file is shown:

using System;
using System.ComponentModel.DataAnnotations;

namespace Project.EntityFramework
{
    [MetadataType(typeof(Dog.DogMetadata))]
    public partial class Dog
    {
        internal sealed class DogMetadata
        {
            #pragma warning disable 0649
            private DogMetadata() { }

            [Editable(false)]
            public object Id { get; set; }

            [Display(Name = "Woofy Name")]
            [Required(ErrorMessage = "Name Is Required", AllowEmptyStrings = false)]
            [MinLength(10)]
            public object Name;

            [Editable(false)]
            public object Collars { get; set; }
        }
    }
}

This Metadata file Dog.Metadata.cs is in the same folder and namespace (folder is not always equal to namespace but by default in visual studio) as the model DatabaseModel.edmx. Metadata files have .metadata in the name by convention.

The fact metadata is being used is declared with the attribute:

[MetadataType(typeof(Dog.DogMetadata))]

Where the type is the metadata class you're using. Therefore it is possible to define the metadata class separately in another file but it's best to keep everything together.

The standard MVC data annotations may used as usual on your metadata class.

Notes:

The constructor for the internal metadata file is private.
The data type of the metadata property or field doesn't matter, you can use object regardless of the underlying type.
It doesn't matter if the target property is defined as a property or field, only the name matters. We've used a property for Id and a field for Name.
If you use fields in your metadata class (for example Name) you will get compiler warnings for the field never being assigned to. You can disable these with #pragma warning disable 0649.

jQuery Fit Footer To Window Height

Eliot (Eliot Jones) — Fri, 08 Aug 2014 00:00:00 GMT

Sometimes your content just doesn't fit the window height, for exampling when you're styling a page with placeholder content. Or, for a short post like this.

That's why I use the script below for making sure my footer appears at the bottom of pages where the content is shorter than the window:

function AdjustWindowHeight()
{
    var requiredHeight = $(window).height() - $("#footer").height();
    $('#wrap').css('min-height', requiredHeight);
}

$(window).on("resize", AdjustWindowHeight);

$(function () {
    AdjustWindowHeight();
});

Or the minified version:

function F(){var a=$(window).height()-$("#footer").height(); $("#wrap").css("min-height",a)}$(window).on("resize",F);$(function(){F()});

All you need is a div containing all your footer content and and wrap div containing all other content:

<body>
<div id="wrap">Content</div>
<div id="footer">Footer :)</div>
</body>

JSON.NET

Eliot (Eliot Jones) — Tue, 05 Aug 2014 00:00:00 GMT

JSON.NET by James Newton-King is the library for working with JSON in .NET. The following is a small guide for using JSON.NET. It is in no way a substitute for the full documentation.

To follow along obtain the JSON.NET package using NuGet and the Newtonsoft.Json dll will be added to your project's references. Alternatively download from the official website and add the reference manually.

Serialize An Object And De-serialize

Serialization is the process of translating data structures or object state into a format that can be stored - Wikipedia.

We will start our investigation with a very simple C# object and continue from there. As always, we are using the Dog class:

public class Dog
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Breed { get; set; }
    public DateTime Birthday { get; set; }

    public int CalculateAgeInDays()
    {
        return (DateTime.Now - Birthday).Days;
    }
}

This is very simple class but might mirror something you need to serialize.

To start, let's create a dog and convert it to a string of Json (which I'll stop capitalising because it's a pain to type).

Dog dog = new Dog
{
    Id = 4,
    Breed = "Labradoodle",
    Name = "Baron Von Lassie",
    Birthday = Convert.ToDateTime("2013-01-07"),
};
string json = JsonConvert.SerializeObject(dog);

The string that we obtain is 'minified':

{"Id":4,"Name":"Baron Von Lassie","Breed":"Labradoodle","Birthday":"2013-01-07T00:00:00"}

As you can see all extraneous whitespace and extra line-breaks have been removed. This is ideal for data transfer objects such as a response from a Web Service, however if we're trying to present our data in a human readable way it's nicer to set Formatting.Indented like so:

string json = JsonConvert.SerializeObject(dog, Formatting.Indented);

This gives the neater response:

{
  "Id": 4,
  "Name": "Baron Von Lassie",
  "Breed": "Labradoodle",
  "Birthday": "2013-01-07T00:00:00"
}

This reveals that more complex datatypes like DateTime are stored as strings in Json. Json.NET uses the ISO standard 8601 for dates, you can read more about why on Scott Hanselman's blog.

We can get our dog back again using:

Dog lassie = JsonConvert.DeserializeObject<Dog>(json);

Serialize A More Complex Object And Serializing To A File

We can now create and retrieve a Json representation of our dog, but how about persisting it somewhere on our hard-drive?

First let's add a list of collars to our Dog object, this will show an object with an object array in Json:

public class Collar
{
    public int Id { get; set; }
    public string Color { get; set; }
    public bool HasTag { get; set; }
}

The collars list property is added to the dog class and the dog now has 2 collars using nice collection initialization:

List<Collar> collars = new List<Collar>
{
    new Collar { Id = 1, Color = "Red", HasTag = true },
    new Collar { Id = 2, Color = "Pink", HasTag = false }
};
dog.Collars = collars;

JsonConvert from the previous example provides a simple way to write and read to/from a string. This provides a wrapper to JsonSerializer.

The simplest way to serialize to a file is shown:

string folder = "C:\\Test\\";
using (StreamWriter file = File.CreateText(folder + "dog.json"))
{
    JsonSerializer serializer = new JsonSerializer();

    serializer.Formatting = Formatting.Indented;

    serializer.Serialize(file, dog);
}

The text file then contains our dog object, preserved forever(!):

{
  "Id": 4,
  "Name": "Baron Von Lassie",
  "Breed": "Labradoodle",
  "Collars": [
    {
      "Id": 1,
      "Color": "Red",
      "HasTag": true
    },
    {
      "Id": 2,
      "Color": "Pink",
      "HasTag": false
    }
  ],
  "Birthday": "2013-01-07T00:00:00"
}

The dog can be retrieved from the file as follows:

using (StreamReader file = File.OpenText(folder + "dog.json"))
{
    JsonSerializer serializer = new JsonSerializer();
    Dog lassie = (Dog)serializer.Deserialize(file, typeof(Dog));
}

Serialization and deserialization can be thought of as hydration/dehydration, dehydrate your dog to store it and stop it rotting, then rehydrate it to play in the park with.

Json Settings

We've already seen one serialization setting, Formatting.Indented in the examples above. There are many more serialization options for which the documentation should be your first port of call.

For the simple Json serializer (JsonConvert) settings are stored in the JsonSerializerSettings object:

JsonSerializerSettings settings = new JsonSerializerSettings
{
    ConstructorHandling = ConstructorHandling.AllowNonPublicDefaultConstructor,
    Culture = CultureInfo.CurrentCulture,
    Formatting = Formatting.Indented,
    MissingMemberHandling = MissingMemberHandling.Error
};
string json = JsonConvert.SerializeObject(dog, settings);

Just out of interest I serialized the JsonSerializerSettings object from above:

{
  "ReferenceLoopHandling": 0,
  "MissingMemberHandling": 1,
  "ObjectCreationHandling": 0,
  "NullValueHandling": 0,
  "DefaultValueHandling": 0,
  "Converters": [],
  "PreserveReferencesHandling": 0,
  "TypeNameHandling": 0,
  "TypeNameAssemblyFormat": 0,
  "ConstructorHandling": 1,
  "ContractResolver": null,
  "ReferenceResolver": null,
  "TraceWriter": null,
  "Binder": null,
  "Error": null,
  "Context": {
    "Context": null,
    "State": 0
  },
  "DateFormatString": "yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK",
  "MaxDepth": null,
  "Formatting": 1,
  "DateFormatHandling": 0,
  "DateTimeZoneHandling": 3,
  "DateParseHandling": 1,
  "FloatFormatHandling": 0,
  "FloatParseHandling": 0,
  "StringEscapeHandling": 0,
  "Culture": "en-GB",
  "CheckAdditionalContent": false
}

Here you can see something else to note, by default Enums are serialized using their integer value, rather than string. For instance Formatting has the value 1 rather than "Indented".

By adding a Converter from Newtonsoft.Json.Converters the string value of an enum can be serialized:

serializer.Converters.Add(new StringEnumConverter());

For the JsonSerializer the settings are properties of the object itself. JsonSerializerSettings can be mapped to a JsonSerializer on initialization JsonSerializer serializer = JsonSerializer.Create(settings);.

One important setting for deserialization is MissingMemberHandling, this controls what happens when an additional property is present in the Json that isn't a property on the target object. If the value is set to MissingMemberHandling.Error a JsonSerializationException is thrown when an extra value in the Json is encountered.

The name for this is slightly confusing since actual missing members (e.g. a property is present on the object but not in the Json) aren't dealt with, the default value of the property (0 for int, false for boolean, etc.) is used.

Json Attributes

A higher level of control over serialization can be achieved by adding attributes to classes. For example, if you only want some properties to be serialized you can add the JsonObject attribute to the class then JsonProperty to each property to include.

For more on this see this page of the documentation.

What Is JSON?

Eliot (Eliot Jones) — Mon, 04 Aug 2014 00:00:00 GMT

This is intended as a simple guide to JSON.

The more programming you do the more you hear about the data format "JSON".

However I've never actually used it until the changes to ASP.NET in vNext encouraged me to use it, basically the old XML format of the Web.Config is out and JSON is in (though you can swap back if I recall correctly).

What's In A Name?

JSON stands for "JavaScript Object Notation".

JavaScript Objects

The origin for this name is clear when you consider JavaScript objects. JavaScript objects are effectively dictionaries of key value pairs.

For those of you not familiar with a dictionary, the concept of JavaScript objects is as follows.

Let's describe your house:

var yourHouse = {
    number : 25,
    numberOfWindows : 20,
    name : "Honeysuckle Cottage",
    dog : new Dog(),
    openDoor : function() {
        alert("Hello!");
        }
};

Now this isn't the best way to express a JavaScript Object with methods (or at all for that matter) but it expresses that JS objects are just collections of names for things and their values.

If I ask, what's the name of your house yourHouse["name"] or yourHouse.name you will give me the string value "HoneySuckle Cottage". If I ask your house number yourHouse["number"] you would give me the integer response 25.

If I open the door to your house yourHouse.openDoor() you will say "Hello!", or more likely, chase me away with a knife.

Similarly if I ask to meet your dog yourHouse["dog"] you wouldn't give me a string response, rather you'd show me the dog object.

Back To JSON

JSON originated from JavaScript objects but has no dependence on JavaScript anymore.

Rather it's a format to express/store/exchange data.

For example the NoSql database MongoDb uses binary JSON (BSON) to store its data.

JSON Syntax Advantages

JSON syntax has similarities with other syntaxes used to store data in a way that both a computer and a person can read, chiefly these are YAML and XML.

One advantage is that it's lighter weight (expresses the same information in fewer characters) than XML, so is increasingly used for Data Transfer Objects from Web Services. The reason for this is if I ask your library web service for all 10, 000 books in the library you want to save every possible byte of your bandwidth.

The other major advantage (or perhaps the main one) is compatibility with JavaScript ("the assembly language of the web"). JSON can be converted to a JavaScript object using eval(), don't do this, due to security, or with a built in JSON parser.

Obviously there are disadvantages as well, have a quick Google to get a more balanced view.

Our House, In The Middle Of Our JSON

Enough talk, time to see some JSON:

{
  "number" : "25",
  "numberOfWindows" : 20,
  "name" : "HoneySuckle Cottage"
}

That's the object representing your house from earlier in JSON. The first thing to notice is your dog is gone, we'll get to that in a moment.

Also concerning, perhaps more concerning depending on how much you like dogs, is that we can no longer open your door. This is because we cannot store functions in JSON, not even JavaScript functions, which are just JS objects. If you can get over the loss of your door, let's get back to your missing dog.

I didn't include your dog because I wanted to show the simplest JSON syntax first, objects can be nested within other objects like so:

{
    "number" : "25",
    "numberOfWindows" : 20,
    "name" : "HoneySuckle Cottage",
    "dog" : {
        "age" : 5,
        "name" : "Baron Von Lassie"
    }
}

In this way complex object containers can be built and stored in an easily readable way.

Syntax Overview

Let's define this a little more formally.

Object

Any object in JSON starts and ends with curly braces '{ }'

An object contains "key" : value pairs where the key is a string identifier for the property of the object.

Value

JSON supports the following Data Types for values:

Number (int, float, double etc all in the same DataType)
String ("see me for more string and syntax rules")
Boolean (true or false unquoted)
null
Array
Object

Array

An array can contain objects:

"LightSources": [
    {
      "Id": 1,
      "Name": "candle"
    },
    {
      "Id": 2,
      "Name": "torch"
    }
  ]

Or other DataTypes, e.g. numbers:

"Times": [
    10.2345,
    15.0
  ]

JSON.NET

"Hmm", you say, "these JSON objects sure look like a good fit for C# objects!".

Well I'm glad you mentioned it, because in the next post we will look at some JSON.NET

POCOs from DataTable

Eliot (Eliot Jones) — Wed, 30 Jul 2014 00:00:00 GMT

Edit: There were a few code errors in the original post, these have now been fixed.

For those of us still using Stored Procedures to retrieve information from a database it's useful to have a quick way to pass the resulting DataSet to a collection of POCOs (Plain Old CLR Objects).

The problem is manual mappings are a pain and if they're spread around your data access logic lead to a maintenance headache. That's why this approach by Ricardo Rodrigues is so appealing. As soon as I came across it I decided to use it for all future data access logic on the application I maintain.

The database for this application is a sprawling nightmare of non-existent constraints, mixed naming conventions and bad data-type choices so his approach was a natural fit.

Original Method

To outline the approach (this is mostly a copy of what's on his site but with a few changes relevant to this post), you use a ColumnMapping attribute:

public class ColumnMapping : Attribute
{
    public string FieldName { get; set; }
}

This attribute is then placed above the properties on a POCO to map the column names of the dataset to the property:

public class Dog
{
    [ColumnMapping(FieldName="Dog_Id")]
    public int Id { get; set; }

    [ColumnMapping(FieldName = "NumLegs")]
    public int NumberOfLegs { get; set; }

    [ColumnMapping(FieldName = "Description")]
    public string Name { get; set; }

    [ColumnMapping(FieldName = "Owner")]
    public bool HasOwner { get; set; }
}

This allows you to compensate for poor life choices on the part of the original db developer by having your POCO properties nicely named.

The mapping is then performed as follows:

public class DbUtil
{
    public List<T> ToObjectList<T>(DataTable dataTable) where T : new()
    {
        List<T> returnList = new List<T>();
        for (int rowIndex = 0; rowIndex < dataTable.Rows.Count; rowIndex++)
        {
            T returnObject = new T();

            for (int colIndex = 0; colIndex < dataTable.Columns.Count; colIndex++)
            {
                DataColumn column = dataTable.Columns[colIndex];

                foreach (PropertyInfo property in returnObject.GetType().GetProperties())
                {
                    String fieldName = ((ColumnMapping)property.GetCustomAttributes(
                                            typeof(ColumnMapping), false)[0]).FieldName;
                    if (fieldName.ToLower() == column.ColumnName.ToLower())
                    {
                        property.SetValue(returnObject,
                            FieldToObject(dataTable.Rows[rowIndex][colIndex], property.PropertyType),
                            null);
                        break;
                    }
                }
            }
            returnList.Add(returnObject);
        }
        return returnList;
    }
}

The FieldToObject method is just a quick and nasty method I chucked together for this demo.

private object FieldToObject(object obj, Type type)
{
    if (type == typeof(int))
    {
        int output = 0;
        int.TryParse(obj.ToString(), out output);
        return output;
    }
    else if (type == typeof(bool))
    {
        try
        {
            return Convert.ToBoolean(obj);
        }
        catch { return false; }
    }
    else
    {
        return obj;
    }
}

I use something similar for the application, it allows you to sanitise your data as you map, for instance some fields which should have a data type of Bit for boolean such as "IsValid" actually have int.
This means you're trying to cast values -1, 0, 1, 2, 3, etc to bool and you need a way to specify the expected behaviour in these cases.

Fine Tuning

Having thought some more about this recently it seems like there's room for performance improvements. So I've put together a thoroughly unscientific test. The DataTable is seeded as follows, trying to match the original database's randomness:

DataTable dataTable = new DataTable("DogsDataTable");
dataTable.Columns.Add("Dog_Id");
dataTable.Columns.Add("NumLegs");
dataTable.Columns.Add("Description");
dataTable.Columns.Add("Owner");

Random rnd = new Random();
for (int i = 0; i < 1000; i++)
{
    DataRow tempRow = dataTable.NewRow();

    bool? owner = true;
    if(i % 3 == 0) owner = false;
    else if (i % 11 == 0) owner = null;

    tempRow["Dog_Id"] = i + 1;
    tempRow["NumLegs"] = rnd.Next(1, 5);
    tempRow["Description"] = "Dog" + i.ToString();
    tempRow["Owner"] = owner;

    dataTable.Rows.Add(tempRow);
}

The DataTable is then parsed using a variety of methods and the time for 500 passes recorded using Stopwatch to measure C# runtime as follows:

DbUtil dbUtil = new DbUtil();
long elapsedMilliseconds = 0;
int passes = 500;
for (int t = 0; t < passes; t++)
{
    Stopwatch stopWatch = Stopwatch.StartNew();
    List<Dog> dogs = dbUtil.ToObjectList<Dog>(dataTable); // This method is swapped.
    stopWatch.Stop();
    elapsedMilliseconds += stopWatch.ElapsedMilliseconds;
}
Console.WriteLine("average milliseconds = " + (elapsedMilliseconds / passes).ToString() 
    + " passes = " + passes.ToString()
    + " total milliseconds = " + elapsedMilliseconds.ToString());

(Note the program is run rather than debugged, debugging adds an extra 320 ms roughly to the runtime)
For Ricardo's method the average time is: 46 ms

Here is a manual mapping, using my same rubbishy casting method to keep it fair.

public List<Dog> ToDogList(DataTable dataTable)
{
    List<Dog> returnList = new List<Dog>();

    for (int rowIndex = 0; rowIndex < dataTable.Rows.Count; rowIndex++)
    {
        returnList.Add(new Dog
        {
            Id = (int)FieldToObject(dataTable.Rows[rowIndex]["Dog_Id"], typeof(int)),
            Name = (string)FieldToObject(dataTable.Rows[rowIndex]["Description"], typeof(string)),
            NumberOfLegs = (int)FieldToObject(dataTable.Rows[rowIndex]["NumLegs"], typeof(int)),
            HasOwner = (bool)FieldToObject(dataTable.Rows[rowIndex]["Owner"], typeof(bool))
        });
    }

    return returnList;
}

This method runs in: 3 ms

Now we take Ricardo's method and try to take advantage of manual mapping using the Contains method to find if a column exists in the DataTable:

public List<T> ToObjectList2<T>(DataTable dataTable) where T : new()
{
    List<T> returnList = new List<T>();
    for (int rowIndex = 0; rowIndex < dataTable.Rows.Count; rowIndex++)
    {
        T returnObject = new T();
        foreach (PropertyInfo property in returnObject.GetType().GetProperties())
        {
            string fieldName = ((ColumnMapping)property.GetCustomAttributes(typeof(ColumnMapping), false)[0]).FieldName;
            if(dataTable.Columns.Contains(fieldName))
            {
                property.SetValue(returnObject,
                    FieldToObject(dataTable.Rows[rowIndex][fieldName], property.PropertyType),
                    null);
            }
        }
        returnList.Add(returnObject);
    }
    return returnList;
}

This runs in: 24 ms

That's pretty good but is it possible to squeeze more performance out?

There's a slight performance benefit (~500 ticks) to declaring the capacity of the object list up-front (as expected due to not needing to grow the list). This is a surprisingly small benefit and shows just how well C# handles list lengths changing.

Final Result

The run-time can be reduced to 4 ms by making the changes below:

A new class is declared inside DbUtil:

private class ExtendedPropertyInfo
{
    public PropertyInfo property { get; set; }
    public string fieldName { get; set; }
}

The method then looks like this:

public List<T> ToObjectList3<T>(DataTable dataTable) where T : new()
{
    List<T> returnList = new List<T>(dataTable.Rows.Count);
    List<ExtendedPropertyInfo> accessibleProperties = new List<ExtendedPropertyInfo>();

    foreach (PropertyInfo property in new T().GetType().GetProperties())
    {
        string fieldName = ((ColumnMapping)property.GetCustomAttributes(typeof(ColumnMapping), false)[0]).FieldName;
        if (dataTable.Columns.Contains(fieldName))
        {
            accessibleProperties.Add(new ExtendedPropertyInfo{
                    fieldName = fieldName,
                    property = property
                });
        }
    }
    for (int rowIndex = 0; rowIndex < dataTable.Rows.Count; rowIndex++)
    {
        T returnObject = new T();
        foreach (ExtendedPropertyInfo eproperty in accessibleProperties)
        {
            eproperty.property.SetValue(returnObject,
                    FieldToObject(dataTable.Rows[rowIndex][eproperty.fieldName], eproperty.property.PropertyType),
                    null);
        }
        returnList.Add(returnObject);
    }
    return returnList;
}

This works by only checking for the existence of columns once, rather than every row and saving a few reflection calls by putting them at the start of the method.

Conclusion

The original method is fast enough that performance benefits here aren't really worth the time taken to investigate, however it's an interesting intellectual exercise.

Create Table SQL Server

Eliot (Eliot Jones) — Tue, 29 Jul 2014 00:00:00 GMT

Most guides on the SQL 'Create Table' command seem to only include the most basic arguments necessary to create a table with a few columns.

They don't go on to detail how to name constraints, create indexes and add foreign keys or similar.

If you're one of the (seemingly very few) people not using code/model-first Entity Framework I've posted this small snippet to help you. It includes how to:

Create and name a primary key.
Create and name a foreign key.
Create and name a default constraint.
Create and name a non-clustered index.
Cascade delete.

It does not detail how to create a composite non-clustered index (a non-clustered index on multiple columns).

So, the whole statement is here (note that having a column with the same name as the table will cause problems in EF database-first, the Question property will be renamed Question1):

CREATE TABLE Dog
        (
                Id              INT IDENTITY(-2147483648, 1) CONSTRAINT PK_Dog_Id PRIMARY KEY NOT NULL,
                Title           NVARCHAR(MAX) NULL,
                Name            NVARCHAR(MAX) NOT NULL,
                OwnerId         INT INDEX IX_Dog_Owner NONCLUSTERED 
                CONSTRAINT  FK_Dog_Owner FOREIGN KEY REFERENCES dbo.Owner(Id) 
                ON DELETE CASCADE,
                UserId          INT NOT NULL INDEX IX_Dog_User NONCLUSTERED 
                CONSTRAINT FK_Users_Dog FOREIGN KEY REFERENCES dbo.AspNetUsers(Id),
                CreatedDate     DATETIME2(0) NOT NULL CONSTRAINT DF_Dog_CreatedDate DEFAULT GETDATE(),
                ModifiedDate    DATETIME2(0) NOT NULL CONSTRAINT DF_Dog_ModifiedDate DEFAULT GETDATE(),
                RowVersion      ROWVERSION
        )

To run through a few things to note:

INT IDENTITY(-2147483648, 1)
There's an interesting debate about whether to use a GUID or integer for a primary key, I've gone with integer since to me it's more readable, but a quick Google will provide more than enough reading to the interested coder. I've seeded this primary key with the minimum value for INT, this means values from -2147483648 to 1 aren't wasted.

NVARCHAR(MAX)
NVARCHAR should ~~generally~~ always be used for text columns because it supports Unicode. VARCHAR supports ASCII.

INDEX IX_Question_QuestionSet NONCLUSTERED
Create non-clustered index on column.

CONSTRAINT FK_QuestionSet_Question FOREIGN KEY REFERENCES dbo.QuestionSet(Id)
Create foreign key on a column.

ON DELETE CASCADE
Further to the foreign key constraint, when you delete a record in the parent table (QuestionSet) specifies that columns referencing this record should also be deleted. The alternative is to error if linked records are present.

DATETIME2(0)
The new DateTime data type replacing deprecated DATETIME (from SQL Server 2008 up). The number in brackets is the 'fractional seconds precision', a fancy sounding way of saying the number of decimal points after the second.

CONSTRAINT DF_Question_ModifiedDate DEFAULT GETDATE()
Named Default Constraint.

RowVersion ROWVERSION NOT NULL
The row version replaces TimeStamp for SQL Server 2008 onwards, NOT NULL is not explicitly required and the column will be Not Nullable. Read all about it.

That's all of it, there's much more that can be done with the CREATE TABLE statement and what I have may not follow best practices but it's a helpful reminder of how everything can be combined in one statement. For instance, who knew about sparse columns?

You sank my battleship

Eliot (Eliot Jones) — Wed, 02 Jul 2014 00:00:00 GMT

So this is probably one of those things that everyone else but me knows and I'm just going to reveal my ignorance, but as a note to myself for future and for those who don't know, here's how you remove the need for battleship debugging (scatter the code with breakpoints and hope you hit something) in Visual Studio.

This is the same as 'Pause On Exceptions' in a Javascript debugger such as that in Google Chrome and other debuggers, but if you are new to debugging and debuggers, it's not immediately obvious.

This means the code automatically breaks when any Exception is thrown and allows you to see the exact moment a problem occurs and the nature of the exception at this point.

How to enable

The following screenshots are from the VS2014 CTP however the steps are the same for most earlier versions, if you can't find the Exceptions window, try ctrl + d, ctrl + e (yes, it's Ctrl + Alt + E in the image but I don't think this is the case for pre-VS13 versions).

So first open the debug menu and find the 'Exceptions' option:

Then make sure Common Language Runtime Errors are set to 'Thrown':

And that's it, no more guessing exactly where an error occurred based on a vague stack trace which may be obfuscated by poorly implemented error handling in the creaking old legacy application you're maintaining.

vNext Setup

Eliot (Eliot Jones) — Sat, 14 Jun 2014 00:00:00 GMT

ASP.NET vNext is coming and despite the lack of fanfare around the announcement, it's a challenge to describe it without resorting to meaningless buzzwords like 'paradigm-shift'. While it's hard to pick which of the awesome features of vNext to celebrate most, to me the top features are definitely:

Code changes without compile (courtesy of the, also hugely exciting, new compiler Roslyn).
Cross platform ASP.NET in collaboration with Mono.
Open Source, the entire vNext project is open source.

While I love code changes without compile as much as the next developer I think points 2 and 3 are the most exciting in terms of how Microsoft develop and release software in the future. The unprecedented level of transparency around these projects finally puts to bed the Microsoft vs. Open Source community disputes.

One of the best server side languages (ok, I use it for a job so I'm biased) is now available for all developers on all platforms, though this might lose Microsoft some licensing costs for Server OSes and Visual Studio licenses it also opens up the majority of servers (which are not running MS OSes) to ASP.NET (not to mention, who on earth chooses not to use Visual Studio if it's available). Scott Hanselman definitely describes all the new features better than I ever could, so head on over to his site to read more.

That's enough about why vNext is great. How do we actually get it up and running? The instructions on the Git Repository are probably the best on how to install.

Install Git

The vNext project is available as a Git repo, so in order to get it you need to have Git installed. Installing Git for Windows is extremely easy, thanks to these guys.

Once you have it installed, navigate to the appropriate directory (use 'cd' to change directory, just like command line) and type git clone https://github.com/aspnet/Home.git

This should clone the repo to the correct directory, in my case I didn't change directory before clone and it copied to C:\Users\[Username]\Home

Install the KPM

KPM is used to manage and install the nuget packages for vNext. Inside the Home folder you should see 'kvminstall.ps1'. This is a Powershell script, Powershell is effectively the Windows command line but better.

Run Powershell, which for me can be found by typing 'Powershell' in the Windows 8 start menu. If you cannot find it the .exe's are in C:\Windows\System32\WindowsPowerShell\v1.0 (for 32 bit) or C:\Windows\SysWOW64\WindowsPowerShell\v1.0 (for 64 bit), I used the 32 bit version. Remember to right click and 'Run As Administrator'.

Once Powershell has started you need to set your execution policy in order for the downloaded kvminstall script to run. To find out what your current Execution Policy is type get-executionpolicy. If it's 'Signed' set it to 'RemoteSigned' by typing set-executionpolicy RemoteSigned.

Once this is done, double-click on the kvminstall.ps1 file and this will run the script. The script creates a new folder in C:\Users\[Username]\.kre containing the 'bin' folder with kvm.cmd and kvm.ps1 in (thus it's fairly simple to set up yourself if you have trouble with kvminstall). It also adds C:\Users\Eliot\.kre\bin to the path.

To add something to the Path go to control panel -> system -> advanced system settings -> environment variables and add C:\Users\Eliot\.kre\bin; to the User Variables -> Path.

Once installed either through the script or manually, open a new Powershell Window or Command Line (as administrator) and type kvm list to make sure 'kvm' is a recognised command. Now you can retrieve the latest version by running kvm upgrade or retrieve a specific version by typing kvm install 0.1-alpha-build-0519 (which was latest at time of writing). This installs files to the .kre\packages folder.

Having installed a version typing kvm list should show some installed versions as shown in this picture:

Type kvm use [version] as shown to swap versions.

Note - I was using BitDefender anti-virus and it stubbornly refused to allow kvm to install some versions, I disabled it when working with kvm and kpm.

Run the KPM and KRE

In the command line or Powershell navigate to C:\Users\[Username]\Home\samples\ConsoleApp and run kpm restore as shown below:

Once this has executed successfully you can now type k run and the Console will log 'Hello World' to screen.

The magic of vNext/Roslyn can be seen here, if you now open Program.cs from C:\Users\[Username]\Home\samples\ConsoleApp and add a new line, for example Console.WriteLine("Hello vNext"); then type k run the change happens straight away, without the need to 'Build' in Visual Studio or any other compile steps.

Happy vNext-ing.

New Site

Eliot (Eliot Jones) — Sat, 07 Jun 2014 00:00:00 GMT

I have changed the entire site to use url routing, but haven't quite finished yet so there's a lot less content than the old site.

hello world

Python Development On Windows

Eliot (Eliot Jones) — Sat, 08 Jun 2013 00:00:00 GMT

For absolute beginners setting up Python on Windows can be a bit daunting. There used to be a great guide for setting up a development environment on the Sad Phaeton Blog which is sadly now defunct. I will try to emulate the brilliance of that guide but if you want another guide there's one here.

Installing Python

There are currently 2 main releases of Python, Python 2.7 and Python 3.3. There are some differences in the language between the two and the support for external libraries is better in 2.7 just by virtue of it being older. We'll use Python 2.7 in this tutorial.

Download it here, I use 32 bit Python (Python 2.7.5 Windows Installer) because the installers provided for third-party libraries tend to look for the 32 bit version. You can trick these installers into using the 64 bit version by editing the Registry but unless you're going to be using huge datasets 32 bit should do for you.

Run the installer using the default settings. I tend to install to the directory (C:/Python27). If we open the Windows command prompt (run "cmd") and type "python" we get the following response.

This tells us that Windows has no idea what a "python" is. To educate it we need to change the environment variables and path.

PATH Environment Variable

So what is the Path? Basically it tells Windows where to look for programs that need to be executed from the command window. The Path is a special type of environment variable.

To find out what's in your path type "echo %PATH%" into the command prompt.

The % signs mark out an environment variable. This type of variable is just a name for some text, so the %PATH% variable refers to a semicolon (;) delimited list of directories on the Windows system. The reason Windows doesn't know what python is is because it's not in the path and it can't find it.

Now we come to the fun part! Setting up your own environment variables (disclaimer: no fun will be had). To find your environment variables go to:

control panel -> system -> advanced system settings -> environment variables

As shown here:

Python Environment Variable

The first thing we're going to do to make life easier is give the location of Python an environment variable. Why do this? When we add Python to the Path text if we hardcode the location and then change that location in the future we'll have to edit the path carefully. By using an environment variable if we change version (to Python 3.3) in future we can just change the environment variable and the system will still work.

To add the environment variable:

Add a new system variable (not user variable though you can if you want) called PY27 (or whatever name you want really but I'm using PY27) with the value being the location of the Python folder "C:\Python27".

After adding this open a NEW cmd window and type "echo %PY27%" (or whichever name you chose) and you should see the name of the directory displayed. You can also move the command prompt into the Python folder by typing "cd %PY27%:

Edit the Path

We still can't access Python by typing "python" in the command prompt, so what have we achieved? Well, nothing yet!

We need to add the new Python Environment Variable (PY27) to the Path. To do so find the Path System Variable and edit it. Add %PY27% after a semicolon ';' separating it from the previous entry.

Now if we type "python" into the command prompt (open a NEW command prompt each time you edit an environment variable) we should start the python shell:

So, we've added an environment variable for the location of Python and added that to our path.

Now to setup easy_install.

easy_install

easy_install is a way to easily install (surprisingly) libraries (bits of code other people have written to make our lives easier).

Download and run the easy_install python script according to these instructions. Check it installed correctly by looking in your Python directory (C:/Python27 for me) at the "Scripts" folder for "easy-install.exe". If you have trouble getting it by this method, the wonderful Christoph Gohlke provides an exe installer for specific versions of Python.

Add easy_install to Path

Assuming you have easy_install.exe in your scripts folder let's add it so we can invoke it from command prompt. Reopen your environment variables and edit the path again to add "%PY27%\Scripts" to the Path variable:

Remember to open a new command prompt after making these changes. Type "easy_install" and you should get an error message:

This occurs because we provided no library name for the program to install.

I realise I've guided you through setting up easy_install but nowadays we tend to prefer 'pip' the Python package manager. So let's install that using easy_install. Type "easy_install pip" and you should see:

You can now install packages using "pip install PACKAGE_NAME" in the command prompt, eg the unit testing library Nose:

Now, the main issue with Python on Windows is that Python libraries tend to have extensions written in the language 'C'. Most operating systems come with a C compiler included. Windows, for no good reason, lacks a C compiler which causes all sorts of errors with our lovely new system, eg for numpy:

I still haven't got to the bottom of this error and it seems to have many suggested fixes. I've so far tried using MinGW but will try Visual Studio when I need to, but for most purposes using the Windows installers (absolute lifesavers) or some combination of easy_install and pip will work. Here "easy_install numpy" works fine: