Advice on updating the ASP .net on Apache appliance

You are probably right, I recall saying something like that. Sorry my posI was probably a bit pedantic. Apologies on that.

A CA signed cert definitely gives a better (public) user experience and remains the best option for a pretty secure connection between unknown parties. I haven't used them myself (I have no need), but I see no reason not to use Let's Encrypt. Actually we are planning to provide an easy way to get Let's Encrypt SSL certs on TurnKey for v14.2.

If you are communicating between known parties then a well constructed self signed cert is almost certainly more secure. The parties must already have trust though (e.g. shared the cert via PGP encrypted email or a local CA). If you were to use a self signed cert purely for security though, you probably wouldn't want to use the default TurnKey cert either!

Regarding the CN in the self signed cert, I have no idea about that. If we can get that working OOTB without any security implications I'm in! I'm assuming turnkey-make-ssl-cert will need tweaking?!

Re comment about the cert.pem being overwritten. The original blog post is 2010 and pre-dates the above script. The comment is valid in context of the original blog post, but should be irrelevant to TurnKe,y at least as of v14.0+. TurnKey now saves it's certs in /etc/ssl/private.

Re CA appliance, yeah I think that'd be a great one. I think Ken has added that to his "possibles" shortlist for when he gets some time free from work.

Ahh, thanks for your explanation of "trimmed down mono appliance". I love your vision. It sounds awesome! For our next major release (v15.x) we are planning on moving to a container format which we have dubbed TKLX. It would really lend itself to what you're talking about. It'd be a lot trickier with the current monolithic builds.

I certainly see the appeal of an asp.net nginx appliance. But with our current builds, every appliance we add adds overhead. That's one of the reasons why almost everything that is primarily a web server uses Apache. Apache is not the fastest or most resource friendly web server. But it is flexible, powerful, well documented and almost uniformly supported by third party software. So basically IMO we chose the best all rounder not necessarily the best for each use case.

Having said that, I'm always happy to hear arguments for and against different ideas. So keep em coming! :)

Jeremy,

Don't worry, I know how busy it can be during migrations and deployments. I know first hand since I have been busy for years with intersecting periods like that.

I am glad that my findings are able to help others who are interested in using .NET with TKL, I will see if I can add more of my findings later.

Regarding my comment about not using the provided self-signed certificate, the comment was not to belittle the awesome work done by the TKL team. As a matter of fact, I am aware of how an unique certificate is generated on first boot, since this behavior made me start looking into the internals of TKL leading me to research more about how it was accomplished.

During my research, I found a post here (I believe it was by you, but after spending about 30 minutes searching for it I have been unable to find it again) which stated that the generated certificate was intended to get the you quickly going but it was encouraged that people would create their own self-signed certificates or get certificates signed by an authority. Perhaps this was with the previous version since the certificates were not using stunnel? I am not certain, this is as far as my memory goes.

While setting up a few virtual machines, I found that I needed to generate my own self-signed certificates since the one included didn't have a CN. Also, it appears that the included certificate might get overwritten when the ca-certificate package is updated. As I had to generate my own self-signed certificates, I kept wondering if there would be an appliance that would allow one of my machines to behave as a Certificate Authority (not yet, but it seems that it is being considered). I also found myself having to execute code on all my machines so that the certificates would be accepted (which also led me to try to get a DNS appliance so I would not have to type the IP of my testing virtual machines... This I am still struggling with a little).

Regardless of that, I believe that I need to clarify that I see two main kinds of production appliances:

• Internal (Intranet)
• External facing (Extranet, Internet)

When I mentioned a "production appliance", I was mainly referring to an "external facing" production appliance, since the URL that is in the generated certificate (asp-net-apache) would not include your intended public URL. A similar argument could be made for an "internal" production appliance if you had several instances of it in your intranet.

Please notice that my intention was not to dismiss self-signed certificates. Yes, they have a few issues but for "internal" production appliances (where you have full control of the devices to implement a way to have the certificates being recognized) there might be nothing more secure than a certificate you signed yourself. For this particular scenario, having an appliance that behaves as a Certificate Authority would be ideal. For "external facing" production appliances, self-signed certificates might not be the best choice since you are dealing with perception and people who are not as tech savvy which could feel suspicious when their browser says that the connection is not secure when they navigate to your "external facing" production site. This would be even worse if you are using an appliance to perform e-commerce, since the loss of trust of a user could cost you losing some existing and potential customers.

Sadly, what you describe as the unsafer option (having an external CA sign your certificates which could result in adding potential back doors in them) is what is perceived as the safer option for the majority of the non-tech savvy people. Unless there was a way to spin an "external facing" production Certificate Authority appliance which would sign the certificates and the majority of the browsers recognized it, an "external facing" production appliance might have no choice but to get an external CA sign their certificates. But, even then.. "Who watches the watchmen"? How would your own CA be different to those receiving your certificates from the others? How can you assure them that they can trust you with not adding potential back doors yourself?

 Anyway, I do appreciate your guidance and I will begin following it and posting the results of what I find. Regarding the "trimmed down mono appliance", I was not referring to an improved asp.net apache appliance (which can also be done, of course) but having a core TKL with only mono installed on it and then have other appliances based on top of it, such as the asp.net apache appliance and potentially an asp.net nginx appliance, too.

The idea of a core mono appliance would allow self-hosted applications to be deployed and executed (Maybe provide with a way for the applications to be registered as services to be started?), while other users could decide if they want to use asp .net with either Apache or Nginx. Nginx might be a good option for those that want to spin up multiple servers as "Web Front Ends" and Nginx serve as the load balancer (I haven't really played to see how such configuration would be achieved, but it is a nice possibility). Additionally, consider using Postgresql as the DBMS since it appears asp .net core includes Postgresql but not Mysql, the problem being that existing users of the asp.net apache appliance who use Mysql could be affected.

Regards!

1. Thanks for the heads up.

2.. Oh...! That's not good!

3. Ah, that was a dirty hack around a weird bug we encountered during the v14.1 build. It sounds like that bug has been fixed now so that line needs to be removed. IIRC I added that line. My bad. I should have commented that it was a dirty hack - to be cleaned up!

4. Yay! It actually resolved your initial issue in the end!

But it still sounds like it's a little broken...

For the record, in an appliance in the library, we wouldn't overlay binaries. We'd build from source at build time and clean up after if there is no better options. Or in odd cases, package it ourselves.

If you want to submit some improvements, you're more than welcome. If you haven't already, I suggest you get yourself a GitHub account (they're free). That's the best way to share code if you've got improvements. If you're unfamiliar with it maybe fixing those broken links (in #1) and doing a pull request - would be a good training mission!? :)

One other tip before I go. When you're developing an appliance, it can sometimes speed up your dev cycle if you test in the root.patched chroot (instead of building to ISO and installing to VM). So in the root of your appliance build code, like this:

make root.patched
fab-chroot build/root.patched

In the chroot none of the services are running so you need to manually start what you need. Ideally stop them before you exit too. So you still need to build to ISO for final testing. But for many things it's much quicker way to find and fix issues. You need to make sure that you aren't building multiple appliances at the same time though. Or have multiple copies of services running in different chroots. E.g. Apache in your chroot will fail if it's also running in another chroot.

Jeremy,

I spent some time reading through the documentation (like I mentioned here) on how to create a new appliance and how to improve and maintain an existing appliance, armed with that knowledge I built core and then built a version of the asp.net apache appliance with a modification to address the SSL issue I detailed above.

There are a few things that I discovered during my journey:

1. The walkthrough wiki page has two links broken to fab/share/product.mk (one in the "makefile" section, the other one in the "plan" section), the URL has "blog" in it, when it should be "blob".
2. The reason why the asp.net examples are not installed in the appliance is because when you try to manually install them you get:

   # apt-get install asp.net-examples
   Reading package lists... Done
   Building dependency tree
   Reading state information... Done
   Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming.
   The following information may help to resolve the situation:

   The following packages have unmet dependencies:
    asp.net-examples : Depends: libmono-sqlite2.0-cil (>= 3.0.6) but it is not going to be installed
                       Depends: libmono-system-data2.0-cil (>= 3.0.6) but it is not going to be installed
                       Depends: libmono-system-web2.0-cil (>= 2.10.3) but it is not going to be installed
                       Depends: mono-xsp2 but it is not going to be installed or
                                mono-apache-server2 but it is not going to be installed or
                                mono-fastcgi-server2 but it is not going to be installed
   E: Unable to correct problems, you have held broken packages.

3. Building the asp .net apache appliance throws an error unless you modify conf.main to remove or comment line 47 (service dbus stop)
4. Building the asp.net apache appliance results in having mono version 4.6.0, which makes patching mono moot (Testing showed that without building mono from source, self-hosted applications work with https)

Below you will find more details about the results of my research, the take aways are summarized in those four points above.

When I built the modified version of the asp.net apache appliance, the quick and dirty approach was to simply use the overlay directory to replace the DLL file produced by building mono from source, which for the current version would be by executing the following commands:

# Patch Mono 4.6.0.245/746756c:
# http://www.mono-project.com/docs/compiling-mono/linux/
# http://stackoverflow.com/questions/25171536/mono-httplistener-client-certificate/36721463
apt-get install patch -y # needed to patch automatically (optional)
apt-get install git autoconf libtool automake build-essential mono-devel gettext libtool-bin -y # needed to build
wget http://download.mono-project.com/sources/mono/mono-4.6.0.245.tar.bz2
echo -e -n "83c83\n< \t\t\t\tssl_stream.AuthenticateAsServer (cert, true, (SslProtocols)ServicePointManager.SecurityProtocol, false);\n---\n> \t\t\t\tssl_stream.AuthenticateAsServer (cert, false, (SslProtocols)ServicePointManager.SecurityProtocol, false);\n" > nocerts.diff
tar -xjf mono-4.6.0.245.tar.bz2
cd mono-4.6.0*
patch ./mcs/class/System/System.Net/HttpConnection.cs ../nocerts.diff
./autogen.sh --prefix=/usr # instead of /.configure
make
make install

After that, I went to my TKLDEV virtual machine and built my own version by executing the following commands (Variables bolded for convenience to be replaced):

# Update asp.net apache:
cd /turnkey/fab/products # cd products also works
git-clone https://github.com/turnkeylinux-apps/asp-net-apache.git <new version name>

cd <new version name>
rm -rf .git # remove git
git-init # initialize a new empty git repository

# Create overlays and copy the file /usr/lib/mono/gac/System/4.0.0.0__b77a5c561934e089/System.dll
mkdir -p overlay/usr/lib/mono/gac/System # Make the hierarchy usr/lib/mono/gac/System
scp -r <username>@<IP of the machine where mono was built>:/ur/lib/mono/gac/System/* overlay/usr/lib/mono/gac/System/ # Copy recursively everything in the System directory
# Delete mdb file:
find overlay/usr/ -type f -name '*.mdb' -delete

# Edit conf.d/main to comment "service dbus stop" out (This is a reminder, no command is provided here to do that)

# Build ISO:
make

# Copy the product.iso to a machine where it can be easily retrieved from:
scp build/product.iso <username>@<IP of the target machine>:<target directory>/<new name>.iso

After a successful test, I was going to inquire if using the overlay approach would really be adequate. After all, for a personal patch it would be fine, but it just didn't feel right to resort to such measures for future versions. I was hoping to discuss about the best approach, since the alternative was to build from source and then having to deal with the clean up (which you have detailed above, but I noticed that building from source results in a larger foot print than if you just install it the way it currently is done in the appliance).

However, feeling pleased with the accomplishment, I decided to build the current version of the asp.net apache appliance and see if going through the process would provide a clue to why the examples were not included. I modified conf.d/main to remove line 47, built the appliance and installed it in a new VM, since the examples were still not showing up, I tried to add them manually and stumbled across the error message reproduced above. Out of curiosity, I tried the test self-hosted application in this appliance and discovered that there were no issues with https: All the work to perform the patch was for nothing, since it was really not necessary at that point.

So, in other words, just building the appliance again fixes the behavior I previously described in this thread. It might still be a good idea to work on a minified version of the appliance (a mono-core appliance) and have other appliances on top of it. Perhaps leaving the asp.net apache appliance as it is while giving the others names based on whatever is decided for the core appliance (For example: mono, mono asp.net, mono asp.net apache, mono asp.net nginx, etc.) What are your thoughts?

Regards!

This would be a pure goldmine for a dotNET user who wanted to follow in your footsteps! Thanks so much for taking the time to share this info. You rock! :)

One quick question/point before I move on to answering your questions though... You state "Remember to never use the provided TKL certificate in a production environment..."

Unless I'm missing something; whilst self-signed certs are pretty sucky (the browser warnings and all that), some might argue that they are (at least theoretically) more secure than "proper" certificates. CA signed certificates, almost certainly have NSA (and other "five-eyes") back doors in them. In that context, a properly configured self-signed certificate (assuming that the person browsing can verify that it is what it says it is) is more secure.

Whilst TurnKey does initially come with a pre-generated certificate; perhaps you weren't aware that TurnKey Linux includes an ssl-generation script. It is used on firstboot to auto generates a unique self signed certificate.

But back to your questions:

How do you "clean up" your appliance from all the tools you had to install on it to compile Mono with your own modifications? (I mean, other than the obvious rm -R mono-4.2.3.4 and rm mono-4.2.3.4.tar.bz2 to delete the files used for the compilation)

Assuming that they were installed with apt (as per your post), then they can be uninstalled that way too, e.g.:

apt-get purge patch autoconf libtool automake build-essential mono-devel gettext libtool-bin

Note that this will also remove all config data. So if you want to remove something but may want to reinstall it later (and would like to keep relevant config) then use 'remove' instead of purge.

Also as a final note, I can't confirm that removing all the packages you installed with apt won't break your updated Mono install. I suggest that you remove them and double check that everything is working as it should...

Is there a way to just get the mono run-time from your "development appliance" to a "production appliance"?

Checkinstall might be a useful method? You'll need to install it first, then for your last line, instead of make install; instead run:

checkinstall -D make install

That should make you a debioan package which you could then install on a separate system. Keep in mind though that whilst convenient, checkinstall packages are not compliant with Debian packaging policy. The main impact of that will be that it won't auto install dependencies. You'll need to manually install the required run dependencies (the build deps are only required for building).

How would you package an appliance that had nothing except for a trimmed-down version of mono which would allow you to run self-hosted applications?

Assuming you mean a "mono-improved" TurnKey style appliance, then I suggest that you start with a fork of the asp-net-apache appliance build code and go to town!

If you need any pointers, please don't hesitate to sing out! :)

How would you "upgrade" to a newer stable version of Mono from the appliance?

When the appliance was first built, we used the version of Mono provided by Debian. There are pros (stable, secure and relatively low maintenance overhead) and cons (old version, sometimes contains usability bugs that are rarely or never patched), but whenever possible it is our preference.

When we were developing v14.0 we discovered that Apache-mod-mono was missing (turns out it broke during the development freeze so was removed). For v14.1 we just installed Apache-mod-mono from upstream (Xamarin). But when we did the v14.1 update, the versions had drifted apart so a full install from Xamarin was our only option.

The upside of all that is that you should be able to update to the latest version of Mono by simply running:

apt-get update && apt-get dist-upgrade