You are here
Ubuntu Landscape Problems data KVM networks and data Programming 

Ubuntu Landscape Problems

I ran into a problem a short while ago with Landscape clients unable to connect to my Landscape server, and just recently figured out what’s going on.

Essentially

PyCurlError: Error 60: server certificate verification failed.

I eventually found what the problem was, and hope to share it with you to save you time.

First off, i really love Landscape.  It’s rough around the edges, isn’t fancy, but it does what i need for my in-house virtual environment test-bed.  Best off all it’s free and well documented and integrated.  And so, i’ve chosen Ubuntu as my gold-master standard of VM deployments, as it’s pretty danged easy to care for during the software development life-cycle, and lets me focus on work, rather than mucking about at the OS level.  KISS is the strategy here, especially when the dev cycle revolves around 12 minute increments of intense brain power, before the kids interrupt me!

Landscape lets you manage a bunch of Ubuntu installs, and keeps them up to date in a single click.  You can also define custom bash scripts to run and graph simple KPIs like CPU/Mem/Disk, and even custom values.   It does what i need, and it keeps my systems humming along and fully updated.  I’ve dedicated a very small VM for Landscape, 2 cores, 4 GB ram, and about 30GB hard drive sparse on a qcow2 image running ontop of a RAID 1 SSD.

I run all my test servers using 16.04 LTS , so when a new vulnerability is addressed with updates, I know i’m a click away from updating all my Landscaped-managed devices.

For some reason, all my ability to direct updates and run scripts just resulted in stalled activities , and intermittently Landscape would show me an error like “10 devices have not contacted landscape in the last 10 minutes”, etc.

The first thing i did was check network connectivity between my hosts and the landscape server.  They were all on the same subnet, had no iptables or ufw blocking the flow at either server end or client end of the flow, and a tcpdump did reveal occasional traffic between them.

I then checked out the log files on the client servers, and finally came up with “tail -f /var/log/landscape/broker.log” which showed me an error that isn’t really found around the google search with too much solution:

root@syslog:/etc/ssl/certs# tail -n 20 /var/log/landscape/broker.log
 headers=headers, cainfo=self._pubkey, curl=curl))
 File "/usr/lib/python2.7/dist-packages/landscape/lib/fetch.py", line 109, in fetch
 raise PyCurlError(e.args[0], e.args[1])
PyCurlError: Error 60: server certificate verification failed. CAfile: /usr/local/share/ca-certificates/landscape_server_ca.crt CRLfile: none
2017-11-05 13:51:22,560 ERROR [MainThread] Message exchange failed: server certificate verification failed. CAfile: /usr/local/share/ca-certificates/landscape_server_ca.crt CRLfile: none
2017-11-05 13:51:22,560 INFO [MainThread] Message exchange failed.
2017-11-05 13:51:22,561 INFO [MainThread] Message exchange completed in 0.19s.
2017-11-05 13:52:22,715 INFO [MainThread] Starting urgent message exchange with https://192.168.168.67/message-system.
2017-11-05 13:52:22,899 ERROR [PoolThread-twisted.internet.reactor-0] Error contacting the server at https://192.168.168.67/message-system.
Traceback (most recent call last):
 File "/usr/lib/python2.7/dist-packages/landscape/broker/transport.py", line 71, in exchange
 message_api)
 File "/usr/lib/python2.7/dist-packages/landscape/broker/transport.py", line 45, in _curl
 headers=headers, cainfo=self._pubkey, curl=curl))
 File "/usr/lib/python2.7/dist-packages/landscape/lib/fetch.py", line 109, in fetch
 raise PyCurlError(e.args[0], e.args[1])
PyCurlError: Error 60: server certificate verification failed. CAfile: /usr/local/share/ca-certificates/landscape_server_ca.crt CRLfile: none
2017-11-05 13:52:22,899 ERROR [MainThread] Message exchange failed: server certificate verification failed. CAfile: /usr/local/share/ca-certificates/landscape_server_ca.crt CRLfile: none
2017-11-05 13:52:22,900 INFO [MainThread] Message exchange failed.
2017-11-05 13:52:22,900 INFO [MainThread] Message exchange completed in 0.19s.

Ha!  This is a problem i’ve done a lot of google searching on, but nothing immediately helpful.  One page mentioned that i needed to modify my /etc/landscape/client.conf file, but this just isn’t the solution here.  If it’s a new install and you’re following some kind of instructions I’m not familiar with, maybe.  This, however, was a different problem with a Landscape installation that was working, and then, suddenly, not working.

So i started to examine my trusted certs on the client system.  Is it possible that my system went through some kind of update where it messed with my cert?

One part of my procedure that i go through when i roll out a new system is to copy the self-signed certificate I generated on the Landscape server over to the client system, and dump it in /usr/local/share/ca-certificates directory, then run the “update-ca-certificates” program.  This usually compiles a giant list of trusted root-CAs, and scrapes in any “*.crt” that sits in the /usr/local/share/ca-certificates dir.  So I ran the program again.  I restarted the landscape-client daemon, and still saw the same Error 60 messages.

My next step was to check and see if there’s a problem with the proessing of the cert.  For this i tried a curl to the URL mentioned in the broker.log file.

root@syslog:~# curl https://192.168.168.67/message-system
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

Neat.  Same type of error, 60.  So i checked into the URL mentioned in the output of the curl script at http://curl.haxx.se/docs/sslcerts.html and saw a handy open ssl command.

root@syslog:~# openssl s_client -connect 192.168.168.67:443
CONNECTED(00000003)
depth=1 C = CA, ST = ON, O = Internet Widgits Pty Ltd, CN = 192.168.168.67
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 192.168.168.67
verify error:num=10:certificate has expired
notAfter=Sep 23 00:00:19 2017 GMT
verify return:1
depth=0 C = AU, ST = Some-State, O = Internet Widgits Pty Ltd, CN = 192.168.168.67
notAfter=Sep 23 00:00:19 2017 GMT
verify return:1

Certificate chain
0 s:/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=192.168.168.67
i:/C=CA/ST=ON/O=Internet Widgits Pty Ltd/CN=192.168.168.67
1 s:/C=CA/ST=ON/O=Internet Widgits Pty Ltd/CN=192.168.168.67
i:/C=CA/ST=ON/O=Internet Widgits Pty Ltd/CN=192.168.168.67

Server certificate
—–BEGIN CERTIFICATE—–
MIIDsjCCApqgAwIBAgIJAJpv5JbGd2EjMA0GCSqGSIb3DQEBCwUAMFYxCzAJBgNV
BAYTAkNBMQswCQYDVQQIDAJPTjEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQ
dHkgTHRkMRcwFQYDVQQDDA4xOTIuMTY4LjE2OC42NzAeFw0xNjA5MjMwMDAwMTla
Fw0xNzA5MjMwMDAwMTlaMF4xCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0
YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQxFzAVBgNVBAMM
DjE5Mi4xNjguMTY4LjY3MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
zYIUGxOZFW0f0J/9eFmxZPdISZq2H4XBv+zyATfpOCR+j7qoZ0nQ7a30atqUz/xQ
mBD75qyGN6l/I0CUSz6reA6/ZdupYTS3bY8L6TPduX30/3aVBGI/HCbKYUfO8CeQ
+P8wJNl+KA/eR+xXcGnfbHESPehU2OyGJAcpxAuFcqxRz+/ZEdei1VGjQPoI0FmE
g1SSrWTqDOVZicaeBfwv8gHZjnhrczj6xKp/uutnzaFsOZIlIGO1VokVIEb1xJdo
VlWyb5enSYkkxtShM/t2KXFKCXW/s+WkKCeX5Cj4R2Oqs/PwtiFR+xbTCrHOQw1G
KPsFgQMdgUySqm56IK10HwIDAQABo3sweTAJBgNVHRMEAjAAMCwGCWCGSAGG+EIB
DQQfFh1PcGVuU1NMIEdlbmVyYXRlZCBDZXJ0aWZpY2F0ZTAdBgNVHQ4EFgQU+hqL
G/DCboQJCQkQ9ZOscKTjPE8wHwYDVR0jBBgwFoAUR74ikDp8kYdblKn7nZEoaIrb
0+cwDQYJKoZIhvcNAQELBQADggEBAHIDIXgtMMi5AEkA4XUWuW93NuTEX+6wVoT6
ZQmLnSLxpnK9AoQDoOI/mDBHzdVd0m2h/ALGwaesmW5XVvYdQ8qQs+L1ts5b9PE+
a+7VErqCe830d3LerypyfT7+ICpIe3HXQfi3a/BL4EiymVRyyV0Y9SrT4fYtvk59
XZQCoYJ5yho/hBo+GxoKLZZp7lzn4/pXrfPynRtVWD3VbZ/J+Hzm7s7rTRKcvURT
9VKjftB0O2N8aTKB9zoMiM62AHJC7iG8l0wD6X9b6zHxqHJ90X1ZL+HoOl38k1d+
GJPRTdNpZSkbyaJQR3j25HFR9xXPVIWFX9EoYrP8g/tPCfy5Dcs=
—–END CERTIFICATE—–
subject=/C=AU/ST=Some-State/O=Internet Widgits Pty Ltd/CN=192.168.168.67
issuer=/C=CA/ST=ON/O=Internet Widgits Pty Ltd/CN=192.168.168.67

No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 2543 bytes and written 431 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: DE04E08F17C20FD9E23371C18B0B01B3B0F607E004ACAEC521799B299D08711F
Session-ID-ctx:
Master-Key: 7F5E4A579E3DF28F66E9B8F3C76A12855E9026DEFEDBAE1F725DEC51BD20E24DDA5D57CF2F83F24988201A7BF23240EB
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 – 96 b4 3e 57 7b 10 4f 2a-37 7d 3f 7e a1 3c 07 9f ..>W{.O*7}?~.<..
0010 – 4e 93 cf 5e ca da d6 b6-c5 d4 e7 50 9c c7 dd fe N..^…….P….
0020 – bb 30 79 ce 63 1f 25 f5-92 6f 81 cc ad f2 09 8d .0y.c.%..o……
0030 – 1c 19 40 ad 85 15 78 57-a5 71 bb 3a 3e d0 9e d0 ..@…xW.q.:>…
0040 – 49 a0 83 20 b1 65 30 db-88 6f 3b 53 d3 ee 8c 6b I.. .e0..o;S…k
0050 – 5f a0 95 c3 5d 1f 00 1d-fc 10 6d 02 65 cc f9 3b _…]…..m.e..;
0060 – cf 13 2d b3 eb 6b 1e 08-78 ea 1f 4f ed 82 7c a6 ..-..k..x..O..|.
0070 – 01 ab 68 4e 7e ab 68 20-37 00 72 ec 22 3f d7 f1 ..hN~.h 7.r.”?..
0080 – ab 4a 07 f3 e2 81 14 fc-34 1a 40 c7 c3 03 72 43 .J……4.@…rC
0090 – 2c ec 91 da 3a 37 55 b8-4b e6 41 59 2c 3b c5 68 ,…:7U.K.AY,;.h
00a0 – cd 32 a9 ad 7a 02 d6 9a-b4 05 06 34 1f 13 65 73 .2..z……4..es
00b0 – 6b 01 2b df e2 5e ce af-f1 00 d8 81 75 55 68 25 k.+..^……uUh%

Start Time: 1509917246
Timeout : 300 (sec)
Verify return code: 10 (certificate has expired)

^C

So there it is.  When i created landscape a year ago and cooked off a certificate, it was set to expire in 1 year.  Dang.  If i had seen it coming, if Landscape had ONLY warned me!  I could have used landscape to deploy the updated cert in batch.  Now i’m going to have to redeploy the updated cert to each one of my machines.  Glad i’m not the kind of guy who has 100 servers running off of Landscape.

So, here’s what i’m going have to do using directions from Canonical on LDS/SSL .  I already have my own CA, so let’s just generate and sign another certificate, this time with a 20 year lifespan!  Revocation is easy, annual unwelcome “reminders”, not so much.

For your reference, here’s my general HOWTO procedure for bringing up new clients in Landscape.

 

  • From the Landscape server, copy /etc/ssl/certs/landscape_server_ca.crt from the LDS server to the client’s directory /usr/local/share/ca-certificates/ and run ‘sudo update-ca-certificates’.
  • (scp /etc/ssl/certs/landscape_server_ca.crt username@192.168.168.67)
  • sudo apt-get install landscape-client
  • sudo landscape-config –computer-title “mynewservername” –account-name standalone –url https://192.168.168.67/message-system –ping-url http://192.168.168.67/ping

 

Related posts

Leave a Comment