2016-08-16

Happy Birthday to Debian and CPAN!

I almost let this day get away from me, but I've taken a moment out of my day to celebrate and reflect on the impact on my professional career that these two systems, Debian and CPAN, have had.

The Debian project first release was announced by Ian Murdock on August 16th, 1993, nearly seven months before the Linux kernel hit version 1.0. Two years later on August 16th, 1995 the Comprehensive Perl Archive Network (CPAN) had it's first upload, Symdump 1.20 by Andreas König.

A couple years after that I was on one of those book club deals where they ship you a book and hope you forget to return it so they can charge you. One such book was about learning Linux and it included a Red Hat installer. At the time I had access to an old x86 computer and a room (a big storage closet kind of room, but it had a window) where I could try installing it. This lead me down the path of PHP (for dynamic web pages I didn't stay long), MySQL (same, but it introduced me to SQL) and GNU/Linux systems in general. I really enjoyed it, enough to pick up an optional UNIX Shell class on campus and checkout books on database design to learn on my own.

These experiences helped me fly through my job interview and have set me sailing on my current course. It wasn't until a couple of years later that I discovered Debian though. Work had a couple of VA Linux systems with Red Hat on them but after maintaining them for a couple of years, some upgrades that insisted on format/install, and manual RPM dependency management (thank-you rpmfind.net for giving me some help) and after trying the Red Hat Network out for a little while before they decided to only deal with the big fish I learned about Debian.

When I read the magic that was apt-get install, I was hooked. When I read testimonials about seamless upgrades and only needing to reboot when the kernel updated I was sold. The switch to Debian 2.0 (Potato, as in Mr. Potato Head from Toy Story) was the last wipe & install I've had to do and I think if I knew enough then, I could have used some slick debootstrap magic. The hardware needed to be replaced by then anyway.

Working with Debian has steeped me in the Perl culture and and CPAN quickly became my magical toolbox of getting things done. Need to talk to a database? DBI has you covered. Want to act like a web browser? Check out LWP. Need to play CGI? Well use CGI (in the beginning.) It wasn't long before we built out our own customer management system in Perl because we could and all the code others shared on CPAN made it possible.

If you contributed in any way to Debian, Perl or CPAN, thank you so much!

Happy Birthday Debian and CPAN!

2015-04-24

StartSSL in Java

Yesterday I had a moment to finally try NetBeans 8.0 against our existing Subversion managed code which I had migrated to an https location with a StartSSL certificate. The web browser and, in my hazy memory of the past, TortoiseSVN clients had had no issue with the new location so I was surprised to run into this error message:

Error validating server certificate for 'https://mysvnrepo.tld:443':
 - The certificate is not issued by a trusted authority. Use the fingerprint to validate the certificate manually!
 I didn't try accepting because that made me think I had something configured incorrectly. My NetBeans 7.3 install was working fine, but it was limping along in CLI mode for Subversion since it isn't updated for the latest Subversion client version to work with my updated working directories.

Some searching around the NetBeans forums lead me to some suggestions for debugging the issue using -Djavax.net.debug=ssl so I whipped up a test application that uses the Java URL class to GET the content of https pages. Accessing sites using GoDaddy certificates worked just fine, but the ones using StartSSL certificates was a no-go.
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
 StartSSL is reasonably new to the CA arena when compared to the likes of Verisign and Thawte, having operated a Certification Authority (CA) since 2005. Their model of operation is appealing, especially to the budget conscious, because you pay for verifications of individuals and organizations, not for issuing of certificates. When you compare their $59.90 fee for a StartSSL Verified status to get access to as many of their class 2/3 certificates as you need to places that charge you $150+/year for each web site certificate and $200+/year for each type of Microsoft Authenticode, Java, and Adobe AIR certificate you start to see why Thawte was worth so much to Verisign who was in turn purchased by Symantec. StartSSL is supported by Microsoft Windows, the major browsers and on mobile devices by having their root certificate included in those browsers and devices.

It isn't included in the Java cacerts file.

To get a CA root certificate added to the cacerts file, a CA is suppose to apply to the Oracle Java Root Certificate program. The startcom / startssl user Admin indicated in 2011 that they have done this with no success. Users have also tried via bug reports to get the certificate included and were rejected saying it must be the CA and not the users that drive CA inclusion.

I think someone has their head on backwards and is facing the wrong crowd. If the developers, the users of your language, are interested in having a CA added then the way they went about requesting it was exactly right. If a bunch of my users or potential users say they will stay with me or start using my product if I add support for API X, there is a strong incentive for me to contact the makers of API X and not wait for them to contact me. I suspect that in the beginning Sun didn't wait for Verisign and Thawte to come ask to be included in their cacerts file. Even Thawte's dead Personal Freemail CA is still in the list and Thawte says it can be dropped in 2011.

Curious what certs are in your Java install's cacerts file? Have keytool tell you. The trick is to tell keytool to list, verbosely, the cacerts keystore who's password by default is changeit. That's a lot of output so you may want to filter it to just the Owner lines.

*nix Shell:
echo 'changeit' | keytool -list -v -keystore $(find $JAVA_HOME -name cacerts) | grep 'Owner:'

Windows Power Shell:
PS C:\Program Files (x86)\Java\jre1.8.0_25\bin> .\keytool -list -v -keystore ..\lib\security\cacerts | select-string -pattern "Owner:"
Enter keystore password:  changeit

What to do?

The real bear is in desktop Java. For our server systems I can add StartSSL to the certificates. For my Subversion issue I can add an exception and if it doesn't stick I can add the StartSSL root to my desktop's cacerts file. It is not very reasonable for anyone but Oracle to add the StartSSL CA root to every end user's cacerts file which puts a damper on using StartSSL to sign Java Web Start applications or applets or access StartSSL signed web sites.

To avoid the Java security code signing restrictions I could switch away from Java Web Start to shipping "executable JAR files", but still I have issues accessing https servers using StartSSL signed certificates even though these sites work fine from the browser and C++/MFC code on windows using the CHttpFile class. I would still need to either add the StartSSL CA root to the cacerts file, disable certificate checking, or avoid the Java URL class and use something else like the Apache Commons HttpClient to dynamically insert trust for StartSSL. Blah.

I could go all-out. Include java.dll and friends and replace javaw(.exe) with MyApp(.exe) that points to my own cacerts file, uses my own icon, and tries harder to act like a native application. No waiting on Oracle. No trying to dynamically modify how certificates are checked. Keep using StartSSL. This would be at the cost of giving up everything Web start and others are doing for me.

Of course I could also stop using StartSSL and switch back to one of the authorities who's root CA key is in Oracle's Java cacerts file, but I wanted to expand the use of HTTPS, not run it at minimal levels. Or I could switch away from Java.

If you care about this issue, maybe we can use social media to raise our voices instead of getting shut down at a bug report for "open"jdk or swept behind the scenes of a CA only apply here forum. Let's see if we can get #startsslinjava trending. Share. Like. Blog. Pass along.

2015-04-23

Yet Another Annoying Password Requirements List

Yesterday I tried creating an account on a hosting provider and my first line go-to program for creating passwords failed to meet their rules:

Password criteria:
  • must be 8-14 characters long
  • starts with a letter
  • include a lower case letter
  • include an upper case letter
  • include a number
  • include a special character (!,@,#,%)
  • does not contain the username
  • not include other special characters
Through 20 years of effort, we've successfully trained everyone to use passwords that are hard for humans to remember, but easy for computers to guess. – xkcd.com/936 Password Strength

My kingdom for the ability to use a passphrase like correct horse battery staple without these silly and seemingly arbitrary extra rules beyond length.

2015-02-20

Non-global learnyounode without much typing

For whatever reason, when you dive into node.js you come across lots of code that tells you to install command-line javascript programs "globally" into /usr/local. Lots of examples say to do this using the sudo command, eg `sudo npm install -g learnyounode` and others say they get messed up doing that so they suggest just changing the ownership of /usr/local to be you.... I get the feeling that most node.js creators and users are working off of Macbooks or something and have a very single-user view of their computer and perhaps play a little loose with security.

This was a big hurdle for me to get over when I first started playing with Node.js on my Ubuntu desktops and Debian servers. It was like the Ruby version thing all over again. Times a thousand. I didn't really want to commit to one-off programs like learnyounode to be stuck in my /usr/local forever. I thought the thing to do was to use node's prefix option but even then I wasn't sure I wanted the prefix/bin files to be in my $PATH all the time.

Fortunately once I learned a bit and got the search terms right I found that others were also trying to solve this dilemma. One of the solutions I liked was using npm run to run scripts in node_modules/.bin. It let me use those binaries locally when I was in the package's folder without committing to them any other time. This appeals to me more than any of the $PATH modifying ones. So, to use nodeschool.io's javascripting or learnyounode interactive modules it was as simple as this:

mkdir -p node/learn
cd node/learn
npm install javascripting
npm install learnyounode
edit package.json
>>> in packages.json
...
"scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "learnyounode": "learnyounode"
},
...
$ npm run learnyounode
I found it tedious after a while to type such a long command. Especially when adding program arguments. So I used a simple alias for that shell instance:

$ alias learnyounode='npm run learnyounode'
$ alias lyn='npm run learnyounode'
You can do one or the other or whatever you like. I decided that even learnyounode was annoying to keep typing so I used lyn.

I really like this solution for working with these interactive programs. I will see what challenges arise as I get more advanced in my node.js and npm usage. I can foresee wanting a "user local" install but still wanting to slip in and out of it. Maybe using a chroot or something.

One thing this method doesn't do is let these package installed binaries like learnyounode and javascripting work from any directory so their directions of "make and change into a new directory" don't work. Instead the "learn" directory with node_modules is where I create all my practice programs.

2015-02-17

LXC multiple personality disorder

I have a couple of server systems running as Linux Containers (LCX) as a test since Debian 6.0 (squeeze). The host system has been upgraded to Debian 7.0 (wheezy) lxc version: 0.8.0-rc1 and things generally work fine but I rebooted the other day and the containers fell to pieces.

After manually stopping the containers (or so I thought) and starting them up again one of the containers was fine. The other, not so much. Connecting to it remotely with the PuTTY ssh client would fail either immediately with "Network error: Connection reset by peer" or it would work for some seemingly random amount of time, a second to minutes, before another error would appear "Network error: Software caused connection abort"

Scouring the web I found lots of suggestions saying it was missing config files or keys in the instance's /etc/ssh/ directory, but I knew this was not the case. The files were there and the connection worked, sometimes. Plus I did some tests running netcat (nc) as a client and a server and those connections also failed either after a while or sometimes right away. Sometimes when connecting to the server instance I had just started I would be told the connection was refused.

I started to believe that I had another server running in my network that claimed the same IP address and server name on login. This belief moved to some kind of server multiple personality disorder when I saw that my tmux session sometimes existed and sometimes didn't on login even though the file I had created in the "tmux exists" connection was there in the "no tmux" connection.

I popped onto the lxc irc channel on freenode for some advice. A fellow user, wam, ran me through some tests. I wasn't running out of memory. My configuration was very similar to the working container. No firewalls were blocking stuff on this internal private network. He suggested that I use tshark to track down the possible RSET, so I went (t)shark fishing:

  4.999284 3com_c0:25:71 -> 46:4d:07:7e:87:9c ARP 60 Who has 192.168.1.33?  Tell 192.168.1.4
  4.999327 46:4d:07:7e:87:9c -> 3com_c0:25:71 ARP 42 192.168.1.33 is at 46:4d:07:7e:87:9c
  5.007975 fa:11:43:eb:f6:eb -> 3com_c0:25:71 ARP 42 Who has 192.168.1.4?  Tell 192.168.1.33 (duplicate use of 192.168.1.33 detected!)
  5.008086 3com_c0:25:71 -> fa:11:43:eb:f6:eb ARP 60 192.168.1.4 is at 00:50:da:c0:25:71 (duplicate use of 192.168.1.33 detected!)

Duplicate use of 192.168.1.33 detected with different mac addresses? I thought I had just ruled out multiple servers.

DeHackEd on the LXC IRC channel, #lxcontaines, suggested checking brctl showmacs which I filtered further using other information he shared:

brctl showmacs br0 | grep -v '  1'
port no mac addr                is local?       ageing timer
  4     46:4d:07:7e:87:9c       no                70.08
  2     fa:11:43:eb:f6:eb       no                22.85
  2     fe:68:85:a8:dc:6e       yes                0.00
  3     fe:b0:fc:d7:2e:9f       yes                0.00

  4     fe:dd:ac:a6:ac:f4       yes                0.00

Both of the systems claiming 192.168.1.33 are running on the LXC host. Odd. Using lxc-ls and lxc-list shows only the working container and the broken one. Not three. Another person, devonblzx suggested that I just specify the hwaddr in the lxc config file. I, in fact, had done this once upon a time and I had long since commented it out. I don't remember why. Maybe it wasn't the unique lxc.network.hwaddr but the non-unique lxc.network.name that was tripping me up. After what DeHackEd and devonblzx pointed out in /sys/class/net/$bridgename/brif/ and /sys/devices/virtual/net/$bridgename/ I bet it was the .name value. I should try it again. The question that nagged me was, how had I launched duplicate instances and would setting hwaddr protect me? DeHackEd said I'm not suppose to be able to launch multiple instances, at least not by the same user, due to control channels that use the names that would conflict. I thought it was maybe a bug in lxc 0.8.0 so I shared my ps output that showed there were indeed three instances with two pointing to the same config file:

root 9287 0.0 0.0 20920 664 ? Ss Feb13 0:00 lxc-start -n lxc -f /etc/lxc/auto/brokencontainer.conf -d
root 18816 0.0 0.0 20920 668 ? Ss Feb13 0:00 lxc-start -d -n brokencontainer

DeHackEd promptly said "no name conflict..." and it took me a minute to spot it. One had been started with the name lxc. I asked why lxc-ls or lxc-list don't show it but no one volunteered that answer so I dove into the start-up process to figure out why it started with -n lxc.

My configuration was from Debian 6.0 worked like this. The /etc/default/lxc file specified both that I wanted it to run from init and listed the CONTAINERS I wanted to start. The init script didn't care anymore about the CONTAINERS variable, instead it looked at /etc/lxc/auto/* and tried deriving the names from them. My /etc/lxc directory looks like this:

/etc/lxc/auto/workingcontainer.conf -> /etc/lxc/workingcontainer.conf
/etc/lxc/auto/brokencontainer.conf -> /etc/lxc/brokencontainer.conf
/etc/lxc/workingcontainer.conf
/etc/lxc/brokencontainer.conf
/etc/lxc/debconf

This is not quite how the README.Debian file suggests things to be:

LXC container can be automatically started on boot. In order to enable this, the LXC init script has to be enabled in /etc/default/lxc and and container that should be automatically started needs its configuration file symlinked (or copied) into the /etc/lxc/auto directory.
Note that the name in /etc/lxc/auto needs to be the container name, e.g.:
  /etc/lxc/auto/www.example.org -> /var/lib/lxc/www.example.org/config
I joined the #debian channel on  the OFTC IRC network to get some advice and figure out if my init.d/lxc script or something was messed up and peter1138 helped straighten me out. He said he had a similar setup when he first upgraded from squeeze but when he created a new container in wheezy he saw the config files were in /var/lib/lxc/containername/config and that the /etc/lxc/auto/containername pointed there.

This made it so that the init script works by extracting containername from the folder holding config. There is nothing in that process that cares what the file in /etc/lxc/auto/* is named, it just better be a symbolic link to a file in a directory who's name is the container name you want. I complained about config files in /var, broken upgrades, and a seemingly misleading emphasis on the auto/ name and how autostart works and was given the bug! challenge.

I think it would be even better if it just read the lxc.utsname from the file as peter1138 suggested, then it could be a symlink or copy to any file without needing some specific directory layout. I said it didn't seem that the name in auto needed to be the container name at all and peter1138 agreed that for autostart to auto start this was true, but if the name was the container name then lxc-list would tag the container in the listing as autostart.

I hope this is helpful to someone else facing similar sounding issues even if that someone else turns out to be a future me.

2015-01-28

IRC SSL Client Certs

ChatZilla supports using SSL connections and auto-identifying with SSL Client Certificates on the OFTC and freenode IRC networks using CAcert WoT User and StartSSL free email verified certificates. You may have trouble using StartSSL verified user certificates. Tested using ChatZilla 0.9.91.1 in Firefox 35.0.1.

2014-05-06

How do I handle fstab mounts under run in Debian Wheezy?

A release goal for Debian 7.0 ("wheezy") was to introduce a new top level directory, /run, and relocate system state information that does not need to persist through a reboot but that may need to be written early or otherwise when the root filesystem is read only. Other distributions are also introducing /run and a proposal has been submitted to include it in the Filesystem Hiearchy Standard (FHS).

This is all fine and well, but it has tripped up automated mounting of /etc/fstab entries under /run (formerly /var/run).

The proposed update to debian-policy says this:
Files and directories residing in '/run' should be stored on a temporary filesystem and not be persistent across a reboot, and hence the presence of files or directories in any of these directories is not guaranteed and 'init.d' scripts must handle this correctly. This will typically amount to creating any required subdirectories dynamically when the 'init.d' script is run, rather than including them in the package and relying on 'dpkg' to create them.
Can I then conclude that /etc/init.d/mountall.sh is not handling /etc/fstab correctly with regard to mounts under /run or that there should be another init.d script to handle the /etc/fstab mounts under /run correctly or did the writers expect that fstab mounts under /run are invalid and all actions under it should be done programmatically by the individual services and generally be fixed-up by their init.d scripts?

2014-04-23

Exploring StartSSL - Automated Registration Email

Reading about the decision to no longer include CACert.org in the Debian ca-certificates package (Debian bug 718434, LWN: Debian and CAcert) I was introduced to StartCom's free certificate offering. As I investigated their site I was both intrigued by the free offering and the Web-of-Trust program idea, and put off by the lack of clear or sometimes conflicting information.

For the impatient, the TL;DR version is this:

  1. Sign up first for a free (class 1) certificate by clicking Sign-up For Free in the top left of the site. Everything else is confusing.
  2. Use an email address that doesn't do grey listing, spam filtering, or anything, and that you have access to the logs on (is this service only for "techies"?)
    1. If you do have grey listing or spam filtering that blocks the web page test so they give you big red text telling you you're all wrong, disable it or at least allow from the names and IP addresses in their SPF record. (yes, I guess this service is only for "techies.")
  3. If the form submits without telling you your mail server is wrong but you don't get an email pretty quick, log out (top right corner icon) and try registering again.

If you'd like to learn more of the details or share my pain, read on:

All paths seemed to lead to getting a certificate so I settled on starting with the StartSSL Free (Class 1) certificate since I wasn't sure exactly what the requirements were to get the StartSSL Verified (Class 2) one. After deciding that "Sign Up" and "Express Lane" are the same thing, and seeing that I must fill out the form as an individual, I entered my personal (gmail) address.

This took me to a page asking for me to check my email right away and copy/paste in the code they sent me. Now Gmail is usually very fast about showing new emails, but nothing was there. Not in Important and unread. Not in Everything else, and not even in the Spam folder. Not several minutes later. The page was very insistent that I do not leave or reload it so in a new tab I started searching for answers.

The first answer I came across can be summarized thus "it must be your problem" with no additional suggestions. I have come to identify this as a common communication style from StartCom:
Important! Experience has shown that the failure of email messages not arriving are always the fault of the receiving end. If the wizard confirms to having sent the message, i.e. no error occurred, than the message has been delivered and accepted by your mail server!
 Surely they've had Gmail users do this process before. So strange that it wouldn't work. After all, I wasn't using one of their blacklisted email providers listed on their enrollment page. I decided to try again from a different browser using my work email address, the one that I manage and have access to the server logs on. This is what I learned.

When you click Continue on the enrollment page your server will get hit from one site. In my case it was [212.117.158.94]. If you have gray listing in place (the work server does) and it sends back an error like 450, the web page immediately tells you it couldn't deliver the email. It does mention that the problem could be grey listing among other things, and basically says it's your fault. So you try to open up your grey listing to allow startcom.org through, but that doesn't seem to be enough because for some reason the client name comes through as unknown. (Edit: I had recently upgraded our mail server and I believe the "unknown" issue was a local configuration issue.)

So you add their IP address and then the web page thinks that all is well and sends you to the "wait for it" code confirmation page, but still no email. Why? Probably because the web page just does a test connection. Right after it sends you to the next page another server, [192.116.242.7] in my case, connects (also with client_name=unknown) and gets Greylisted. So I sit here waiting, hoping for a retry, feeling stuck with no help. Back to searching in a new tab.

The second answer I came across also says "it must be your problem" :(
The program always sends the verification code! Do not blame us, if it does not arrive....we do not have control over your mail server and mail account!
 Third time's the charm? Good thing I have three browsers installed. So I checked the SPF (TXT) record for startcom.org and added all of the names and IP addresses listed into my server's client whitelist for greylisting and tried again from the third browser using the work email address. Success! The email made it to my inbox.

I didn't really want to do the certificate in the third-choice browser, so I went to the second browser and pasted the code there. It failed to verify but the failure message told me something I would have loved to have known long before. I didn't copy the exact message, sorry, but basically it said "if it fails, log out and try to sign in again". A "resend this request" button would have been better, but at least now I know that I don't have to stand like a deer in the headlights on the "wait for it" page when things fail.

Now I just have to wait 6 hours for the account to be reviewed, probably because I tried so many times.

Good luck. I may end up dabbling with CACert, Comodo, or retreating to my own self-signed certificates again.

2014-01-18

FamilySearch Indexing on Ubuntu 13.10 x86_64 via Oracle JRE 7

The hard disk drive on my old used Toshiba Techra A9 suddenly died a death of a thousand bad blocks, so I replaced it with a SSD (snappy!) and re-installed Ubuntu. They seemed to be pushing 64 bit unlike a couple of years ago, so I went with that since it was supported, not because I have oodles of bios hidden RAM.

The next thing to do was install some of the programs I use the most. NetBeans and Eclipse are right up there, so first stop was Java. I like the idea behind OpenJDK and recognize the push to use it, but I've been bitten many times in the past by performance and display, and outright broken issues (looking at you web start) that I go straight to the Oracle JDK. Sorry guys. I used the webupd8.org ppa to install it. Good stuff. One less thing to manage in /opt. Next came NetBeans, Eclipse, and the Arduino IDE, all Java based.

Since I was on a roll with Java based programs, I thought I'd stick FamilySearch Indexing back on. The Eclipse "unzip it where you want it" and the Netbeans and Arduino installers had gone well enough that I didn't expect any trouble, but that's what I got.
bin/unpack200: not found
After a few false starts, I found the best answer for this issue on the LDSTech forums where I was put onto the idea that it was a 32 bit compatibility issue on some 64 bit setups, and the work-around was to install the ia32-libs package. So I tried that:
Package ia32-libs is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  lib32z1 lib32ncurses5 lib32bz2-1.0

Reading around I found more confirmation on the ia32-libs package going away. This seemed like a roadblock, except that the error had to do with decompressing. Maybe ia32-libs only included those three packages, so I tried the first, lib32z1, and that error went away, but another appeared.

java.lang.NoClassDefFoundError: java.awt.Container
    at com.install4j.runtime.installer.frontend.headless.AbstractHeadlessScreenExecutor.init(Unknown Source)
    at com.install4j.runtime.installer.frontend.headless.ConsoleScreenExecutor.(Unknown Source)
    at com.install4j.runtime.installer.frontend.headless.InstallerConsoleScreenExecutor.(Unknown Source)
    at com.install4j.runtime.installer.Installer.getScreenExecutor(Unknown Source)
    at com.install4j.runtime.installer.Installer.runInProcess(Unknown Source)
    at com.install4j.runtime.installer.Installer.main(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.exe4j.runtime.LauncherEngine.launch(Unknown Source)
    at com.install4j.runtime.launcher.Launcher.main(Unknown Source)
How could java.awt.Container not be defined. This now sounded like a Java Runtime Environment (JRE) issue, but I know mine is fine. I just installed and tested three IDEs. It wasn't until this point that I noticed it was an install4j based install, so I started including that in my searches.

The problem wasn't unique to FamilySearch Indexing. I found someone trying to troubleshoot it for Visual Paradigm for UML among other things. Their solutions were the now obsolete ia32-libs or making sure their installed JRE was good.

Then I came across a post by Matthew O. Smith talking about Indexing and obsolete ia32-libs. He installed a number of extra libraries, including some i386 ones, and then installed using the headless option to get things going and save on installing a few more libraries. I felt like I had gone far enough with lib32z1 so I decided to try a different route, to try and run the Indexing software with my Oracle Java 7 environment. To do that I first leveraged the headless install tip Matthew gave, and then I modified a copy of the install4j shell script indexing launcher it created in $HOME/.FamilySearchIndexing/indexing.familysearch.org/
./Indexing_unix.sh -J-Djava.awt.headless=true
$ diff indexing indexing-jre7 
4c4
< # INSTALL4J_JAVA_HOME_OVERRIDE=
---
> INSTALL4J_JAVA_HOME_OVERRIDE=/usr/lib/jvm/java-7-oracle
114c114
<     if [ "$ver_minor" -gt "6" ]; then
---
>     if [ "$ver_minor" -gt "7" ]; then
I copied the .desktop entry and tweaked it to point to my modified launcher instead and now I'm in business again.

2013-11-30

Java RESTful Web Services, NetBeans Style

I’ve been interested in exploring Java RESTful Web Services to backend some AngularJS front-ends, with my current focus on JAX-RS implementations.

Blaise Doughan has been blogging a lot about EclipseLink, JAXB, and MOXy. I decided to follow the code example in his post MOXy is the New Default JSON-Binding Provider in GlassFish 4 using NetBeans 7.4 since the Java EE download bundles an install of GlassFish Server Open Source Edition 4.0.

Start by creating a new Java Web Application by choosing New Project from the File menu, going to the Java Web category and selecting Web Application. To keep things the same as his example, name it CustomerResource. Select the GlassFish Server 4.0, Java EE 7 Web, with the suggested context path of /CustomerResource. If you run this right away you should be served the index.html page saying “TODO write content”.

We will work backwards a bit in his blog post, building a little infrastructure before we use it. So first we will right-click Source Packages and add a new Java Class named PhoneNumber in the org.example.model package. Paste or type his code into this class. Do the same for the Customer class. NetBeans will suggest you use the diamond inference and make phoneNumbers final. The code works fine either way.

Still working up in the blog, we will create the CustomerApplication and CustomerService classes in the org.example.service namespace. At this point you should be able to click run and visit the local URL to get our “hello world” type xml response for Jane Doe:

http://localhost:8080/CustomerResource/rest/customers/1

Everything up to this point “just works” in the excellent NetBeans IDE and GlassFish Server, but I was interested in his JSON tweaks, having seen some of the shortcomings he mentions unless I map to a JSON object by hand. To do some testing I first commented out the APPLICATION_XML line from the @Produces list so that I could see (download) the output and move forward to Customizing the JSON-Binding with it’s use of MoxyJsonConfig. This is where I was stumped for a bit.

Pulling in the JAXBContextProperties wasn’t a big deal. The EclipseLink from GlassFish library seemed to have what I was after. Just right-click the Libraries folder in the project and Add Library then choose that library and click Add Library.

To get MoxyJsonConfig, download jersey-media-moxy-2.4.1.jar and stick it someplace handy. I use a folder named Libraries in my NetBeansProjects folder. Then right-click the Libraries folder in the CustomerResource project in NetBeans and click Create in the Add Library dialog. Name it something like Jersey Media Moxy and then in the library classpath Add Jar/Folder to add jersey-media-moxy-2.4.1.jar. Then add this library to your project.

At this point you should have output like Blaise has documented for New Response in his blog post. Enjoy.

If you can’t find jersey-media-moxy-2.4.1.jar or the API has switched around again and a later version is missing the dependency, then read on for my tale of woe and sorrow trying to locate it in the first place. Perhaps it will help.

I haven’t jumped on board with maven yet, so when I came across Blaise’s follow-up question to the StackOverflow question Cannot import EclipseLink MOXy while searching for MoxyJsonConfig where he implied the use of Maven, I was a little disappointed. I was equally disappointed in my next dozen searches all failing to find the jar containing MoxyJsonConfig. I could find API docs, people talking about using it, etc. findjar.com failed. Even mvnrepository.com searches failed. Finally after Google searches of varying portions of the class or package name, one for org.glassfish.jersey.moxy pointed me to jersey-media-moxy within MVN. Unfortunately it was pointing me to 2.0-m07 which has MoxyJsonConfiguration and not MoxyJsonConfig. I didn’t realize that right away and tried implementing using it. It doesn’t work. The current latest version, 2.4.1, has MoxyJsonConfig and does work. I have no idea when things changed or what version Blaise used.

I was glad to finally find the jar, but there has got to be some better way to find a class and know what version of things people are talking about. If there is, please share. If there isn’t, please keep this in mind when sharing code examples.

2011-01-01

Scratch - programming in disguise

My memory of my introduction to programming is a little fuzzy, but a few notable exposures stand out. In no particular order they were ChipWits, TurboTurtle/Logo, World Builder and a BASIC based program that controlled a little submarine which it seems was tongue-in-cheek called GoSub. Each had it's own reason for sparking some interest in figuring out the logic of a program and each had it's quirks.

I think I spent the most time with ChipWits, an old version for the early Macintosh (Were we up to a "512"(K) or a Plus by this time? I am not sure.) It's simplistic IBOL interface for graphically coding the actions of the little computer on roller skates was easy to grasp, and the sounds and animations as it touched, tasted, ate and zapped (or got zapped) were entertaining.

The TurboTurtle intro was a brief after-class exercise while doing some kind of introduction to computers evening class. I mostly remember pen up, pen down, and seeing the computer screen draw patterns like you could make with a spiral sketch tool.

World Builder started out as a game to play (playing games created with it) but then I wanted to try and make my own story/game. Even though it inspired me to think a lot about the story and programming, I never got more than a basic grasp of the language as I tried learning from other code without a manual. Another hurdle for me was not being particularly artsy so the graphical aspect of the game was pretty lacking.

The "GoSub" program was fun. It seems like it was a program/IDE and a book to teach some programming. I was typing in some actual code that was calling all the drawing routines and it felt like quite an accomplishment when I could actually get the sub to dive and surface and fire a torpedo. The program would have one of those crushing sad mac error messages when the torpedo would hit the target. After a while of not figuring that out I gave up on it.

Looking back and summarizing my experiences, my spark of interest to learn how to program the computer wasn't started by looking at code and thinking "wow, 'declare i integer' looks really amazing. I want to type that all day." No, It was the desire to create something fun that I could then play with or show off to others. Programming was the necessary evil for my desired end.

I want to provide the opportunity for the same spark to code within my own children. Sure, they may not be interested in being programmers, but they will be interacting with computers a lot and it can be helpful to "know how the engine works". Besides, the problem solving and logic skills can be applied to many aspects of life.

At first I thought I could take something like GNU Robots and wrap it in a ChipWits like interface. That turned out to not be a quick task and lead me down the road of exploring other robot programming concepts, most frequently the "robot battle" and even LEGO Mindstorm NXT. It seemed a little too advanced to keep their interest for now, so I looked some more.

There is a lot of buzz about a couple of drag-and-drop programming systems, Scratch and Alice, for introducing programming concepts and empowering people to be creative with their computers.

After watching some videos about Scratch and even more impressively, downloading and playing community contributed programs I'm hoping that my more creatively minded kids will catch some of the vision of how they could use Scratch to express themselves through animated comics (even better than my old friend Comic Strip Factory,) music videos, drawing random patterns (TurboTurtle), and arcade (GoSub) or role playing style games l(World Builder).

The Scratch community gives a lot of opportunity for sharing your strengths to build onto a started project (remixing) or join a team and to show off your works. I think that is a very powerful aspect.

If they push the limits of the Scratch 2d world and are willing to get a little more technical to go 3d, they can step up to Alice and do "Word Builder in 3d". Who knows, maybe one of them will really get into it and be willing to work through How To Design Programs and/or Python Programming: An Introduction to Computer Science or some Python tutorials. Maybe after 10 years they will start to be a really good programmer.

The trick is to get them started and keep it fun and consistent and the best way to do that seems to be for me to do it with them.

2010-01-08

Rethinking Nagios config layouts

I've been using Nagios 1.4 in Debian Etch for some time now. By default they set you up with a working layout like this:

nagios.cfg:
...
cfg_file=/etc/nagios/contactgroups.cfg
cfg_file=/etc/nagios/contacts.cfg
cfg_file=/etc/nagios/dependencies.cfg
cfg_file=/etc/nagios/escalations.cfg
cfg_file=/etc/nagios/hostgroups.cfg
cfg_file=/etc/nagios/hosts.cfg
cfg_file=/etc/nagios/services.cfg
cfg_file=/etc/nagios/timeperiods.cfg
...

The services.cfg file started out with a generic-service template where you set the standard options most generic-service services will be using, and then a list of services. A service definition must contain, among other things, the name of the host or host_group that the service runs on. So each time I setup a new host, I would enter it in the hosts.cfg, hostgroups.cfg, and add all of it's services in services.cfg. I was using hostgroup objects just for display and contact_groups and not taking advantage of them in service objects. Those two things combined made for a long services.cfg file. At least I was taking advantage of a couple of service object templates.

The nagios3 config layout shipped with Debian 5.0 (Lenny) opened my eyes to a new structure. It uses the cfg_dir directive to point to /etc/nagios3/conf.d and in there we have:

contacts_nagios2.cfg
extinfo_nagios2.cfg
generic-host_nagios2.cfg
generic-service_nagios2.cfg
host-gateway_nagios3.cfg
hostgroups_nagios2.cfg
localhost_nagios2.cfg
services_nagios2.cfg
timeperiods_nagios2.cfg

In the localhost_nagios2.cfg file they define a simple host that inherits from geric-host and a few extra services, rather than putting localhost entries into the services config file. Checking out services_nagios2.cfg I found a bunch of generic (eg check_http, check_ssh, check_ping) services using host_group names instead of hosts.

Wandering over to hostgroups_nagios2.cfg I found a bunch of very simple hostgroup object definitions. Some like http-servers listed localhost as one of their members. If we reverse that situaion though and drop the hostgroup.members declaration and instead set host.hostgroups to include http-servers, then if I need to make customizations to localhost or delete it alltogether, I do it all in/to localhost_nagios2.cfg. One file. I like this much better!

Nagios' object based configuration system is very flexible. I don't think any of this stuff is new to version 3. I've read the docs dozens of times but it took a fresh install with a new layout idea, some mistakes on my part and some help from Marc on nagios-users to get it sorted out in my head and on disk. Once I had that, the migration of other hosts to the nagios3 setup has been fast and minimal.

2009-12-01

Threshold filter & what about RDF Annotations

I hope Google Sidewiki has a good threshold filter so we can vote down drivel and not have to see it. Browsing through the first several pages of the Google Sidewiki Sidewiki showed me this neat tool is already getting messy on a high-traffic page, and here I am adding to it. ;)

I've been waiting to try a mature RDF Annotations solution as proposed in the Amaya browser documentation for Annotations. A kind of mix between Google Sidewiki and Google Notebook.

in reference to: Google Sidewiki (view on Google Sidewiki)

2009-06-15

Linux VServer & PAM limits

I've been getting some setrlimit "Security Events" reported by logcheck the past few weeks from a Debian 4.0 (Etch) virtual host running on a Debian 5.0 (Lenny) host. I was busy when they started and didn't drill down to see if it started with a security update in one system or the other.

Jun 15 10:09:01 green pam_limits[4284]: setrlimit limit #11 to soft=-1, hard=-1 failed: Operation not permitted; uid=0 euid=0 Jun 15 10:09:01 green pam_limits[4284]: setrlimit limit #12 to soft=-1, hard=-1 failed: Operation not permitted; uid=0 euid=0

Today I did some searching and found a few related posts ranging from bugs in sshd's limit setting to Linux-Vserver specific cases.

  • http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=171673
  • http://www.mail-archive.com/vserver@list.linux-vserver.org/msg08783.html
  • http://osdir.com/ml/linux.pam/2007-09/msg00024.html

One suggestion from the Linux-Vserver list was to just comment out the use of pam_limit in the pam configuration files. I didn't want to throw the baby out with the bath water so I decided to take their other suggestion and modify the /etc/security/limits.conf file.

First I had to find out what limit #11 and #12 were on my system. I'm running a 686 kernel so I looked in include/asm-generic/resource.h of the matching kernel source. There I found that 11 was RLIMIT_SIGPENDING and 12 was RLIMIT_MSGQUEUE. Before getting there I had grepped around and noticed that the alpha and mips architectures swap a few values around so you may need to look at more than the generic header.

Next I used the ulimit command to list the values of pending signals (-i) and POSIX message queues (-q) applied to a default user session in the host system. I modified the /etc/security/limits.conf to set the hard and soft limits for these two resources to match. This stopped the setrlimit messages in /var/log/auth.log for new ssh logins, but they remained when cron attempts to run as root even after restarting cron. The same happens if I allow root ssh logins and login as root.

  • http://linux.derkeiler.com/Mailing-Lists/Debian/2008-08/msg01096.html

2009-03-19

PuTTY X11 forwarding stopped working on Vista

At work my primary desktop system is running Windows Vista Business 64 and our servers are all Debian. Almost daily I access these servers using ssh via the awesome PuTTY (0.60) program. When I want some graphical display returned, like a gvim, pgadmin3 or etherape, I fire up Cygwin/X and use it as my local X server. Once configured this has been working flawlessly for years on MS XP and Vista the past several months.

Yesterday when I tried firing up etherape to track down some network usage, I got the discouraging message "X connection to localhost:10.0 broken (explicit kill or server shutdown)." I followed some basic ssh and X forwarding troubleshooting steps and everything checked out. Finally I found and installed iftop which was everything I wanted ntop to be and did a good enough job.

Today I tackled the issue and found that a Cygwin/X xterm would properly bring the xclient home to my desktop, so the issue was not the servers (which had not had any software changes in the previous few days) it must be the software running on Windows.

I looked at the PuTTY > Connection > SSH > X11 settings and noticed the X display location was blank. I thought this might be akin to the $DISPLAY variable so I checked the help file. It said that if the field was blank PuTTY tried to use :0. Sounded reasonable, but I decided to test it anyway. The xterm had said $DISPLAY was 127.0.0.1:0.0 so, prefering names over numbers, I tried localhost:0.0. No dice. I then tried 127.0.0.1:0.0 and was rewarded with xclock and friends from the remote system.

A little googling lead me to a bug report from 2004 about possibly needing to specify the X display location. I don't think my issue was the same. The note claims that using localhost:0 on MS Windows should work, and it does not for me. I believe that some recent Windows Vista update or setting change (I have not changed networking settings for months) has broken localhost:0, but somehow using the ip address instead of the name works from PuTTY. It doesn't matter if I use the number or the name in a local Cygwin xterm.

Next I pulled up the trusty cmd.exe and tried ping localhost. This showed me that localhost resolves to the IPv6 address ::1. Perhaps this has changed recently due to a Windows update or some setting change I don't remember? I don't know if the root issue is Cygwin/X server not listening on IPv6 addresses (perhaps my version isn't even capable) or if the IPv6 syntax doesn't play nice with the final :0. I did try setting the display to ::1:0 in the xterm and that didn't work. It said "_X11TransSocketINETConnect: Can't get address for :: Error: Can't open display: ::1:0".

For now I'm going with updating all my PuTTY session configs to use 127.0.0.1:0.0

2008-11-13

LARTC the Debian Way (tm) - Intro

I really appreciate the flexibility and plethora of information you can find on open source topics, especially high visibility items like the Linux Kernel and tools to configure it's run-time state. One collection of tools that has helped me find creative solutions to a number of networking challenges has been the set described in the Linux Advanced Routing & Traffic Control HOWTO (LARTC). What gems the iproute2 (sometimes just iproute) package holds. After I've gone through and whipped up a solution via the ip and tc commands, I start to wonder about how to get it to survive through a reboot. The servers I work with are primarily running recent versions of Debian so when I come across and implement solutions like the custom script I came across at minigeek.org I wonder how to set up these things in "the Debian Way" (tm). I'll make follow-up posts as I find solutions.

2006-08-17

Limited User - The Anti-Virus

A co-worker has been complaining the last week and a half about how frustrating it was to have to re-install their home MS Windows XP Media Center computer because it had a virus. (That reminds me of a cute commercial.) His theory on the attack vector is that his sister-in-law had logged in as administrator to install some software from the Internet.

However it got there, it appeared to have been introduced to the system with admin privileges and was able to quickly zap any anti-virus tool thrown at it. There may have been a way to recover without re-installing, but that's not my point. My point is that once the malicious code had been run with administrative privileges, it was very effective and difficult to remove for many reasons, including it's ability to disable anti-virus software.

I grew up using Macintosh computers from OS 6.x to 7.6 and during that era of mostly Internet free use the main attack vector of viruses were floppies from untrusted sources. The solution was simple, have anti-virus software installed that scans your disk when it's inserted. With the exception of kids deleting your files or messing up your desktop, it wasn't that big of a deal that you were basically always an administrator on the system. The same could be said for MS Windows during the same era. Most of us have learned to use single-user computer systems with virtually no restrictions to what we could do at any time. Multi-user systems were something that businesses and universities did.

Then came OS X (with it's Unix heritage) and XP, multi-user for the masses. Perhaps to help ease the transition from the single-user I can do anything whenever I want experience to the multi-user setup, MS XP Home Edition would by default create it's first user account as an Administrator and require that at least one user account be an admin account. This default setup lead me to believe for quite some time that running XP as an admin was encouraged by Microsoft. I have since discovered that this is not true. In fact, some people at Microsoft go out of their way to write about not running as Administrator, or at least reducing privileges of Internet applications. Aaron Margosis went as far as to claim what I've been wondering (and experiencing) for over a year, that you are better off running as non-admin WITHOUT anti-virus than you are running as admin WITH anti-virus.

Over a year ago I had switched my wife's account over to an Administrator account because she was calling me at work weekly for help logging in as admin to install the latest kids software from the library. (Future rant topic, kids games that need to be installed.) She only browsed to safe sites and the kids aren't old enough to go finding warez so all was well until she opened an attachment sent by a virus on her brother's computer. Fortunately Stinger was able to remove it and I set McAfee up on guard duty.

A few months later my teenage sister came to visit for a week. She wanted to install a few games and I had McAfee, so I made her account an Administrator account. A couple of days later I used the computer and found that IE had an annoying browser helper app. My sister said she didn't install it, but that it just appeared when she went to some site. McAfee didn't protect against it so I ran the gauntlet of anti-malware software. None of them could remove it. So, I re-installed and everyone was a limited user. I asked my sister to browse to the sites she had been to before the trouble started. At one of the sites the browser helper appeared. I logged her out and logged into my account. No helper app. The damage was isolated to her account which was easily deleted and re-created.

For the rest of the year's subscription for McAfee I dealt with their software's annoying borderline LUA bugs (having to Graphically log in as an Administrator to update, multiple update notification processes running instead of installing as a system service, etc) and didn't see it find or protect us against one virus, so I didn't renew. I just run ClamWin occasionally to see what kinds of junk has accumulated in the IE Temporary Internet Files. (I personally browse with Firefox but the kids often use IE.)

At work most people run with Administrator privileges and complain about their anti-virus software pestering them to death with questions and announcements and eating up their system resources. I run as limited user with Firefox, Thunderbird, text-only email and only installing software from trusted sources. Guess who doesn't get viruses (again, I scan occasionally with clamwin runas admin to see what's there).

I would love to see a comprehensive test of what viruses are blocked or limited by running as a limited user on XP. Mike L. at gnuman.com has written about a small test. If you know of such a listing or have done the testing, please let me know about it.

I Am

OK. I've finally given in and created a blog so that I could rant about things in a socially accepted (and popular) forum.

Up to this point I have had no interest in blogging. I'm not very consistent at anything over the long term, especially something journalish and I was more interested in publishing more of a technical writing than an opinion column.

First I did "content management systems" like PHP-Nuke, Post-Nuke and Drupal. Those have their place, but they feel empty without an active community.

Next I got into writing up my insights into a third person form on a wiki for others to learn from or improve upon (edit). I still appreciate wiki sites, but I didn't want to rant on one just to have someone edit my opinions so here I am.

I don't know what this blog will become. The odds are that it will be a neglected collection of sporadic rants of things that annoyed me to the point of writing. I am a code hacking (white hat) open source fan who daylights as a computer programmer and GNU / Linux systems and network administrator with a Windows XP desktop. Most of my thoughts will likely be focused around those topics.

I've gone to the blogspot/blogger (and other blog hosting site) signup pages before when I've had something I felt like ranting about, but picking a name has been a bit of a road block. As I was thinking about what to name this blog, I thought "why do people blog, anyway". Some do it for popularity, and others for a little money (if they are really popular and have advertising, or are promoting something) but mostly I think we do it to say "hey, I am here."

I finally committed to blogging and picked a name with the thought "Cogito, ergo sum" (I think, therefore I am.) unfortunately even though iam.blogspot.com isn't a site, they wouldn't let me have that URL, so permalinks will be to iam-jla.blogspot.com