Nova VNC flows under the hood
Most OpenStack deployments has a VNC console implemented with nova-novncproxy. This service gives the final user the ability to log into their instances in a web based method through a browser.
At this post i'm going to show how a vnc console request works under the hood while using the following command or lauching a vnc session through Horizon.
# nova get-vnc-console INSTANCE novnc
First of all, a user connects to NOVA and issues a VNC console request for an instance. Nova API needs to validate the user issuing an authentication request to keystone.
The user receives a token with nova's endpoint URL in the catalog, with that endpoint and the token, the user makes a request against nova calling for a VNC session.
GET http://192.168.200.208:5000/v2.0 -H "Accept: application/json" -H \
"User-Agent: python-keystoneclient"
GET http://192.168.200.208:8774/v2/ -H "User-Agent: python-novaclient" -H \
"Accept: application/json" -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1"
If a name instead of an instance ID is used in the request, Nova need to check his database to match that name with his corresponding ID, as we can see in the following request.
GET http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers?name=test1 \
-H "User-Agent: python-novaclient" -H "Accept: application/json" \
-H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1"
RESP BODY: {"servers": [{"id": "9165dbda-f54e-4186-b2cb-e6ca05ac53ee", \
"links": [{"href": "http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "self"},\
{"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", \
"rel": "bookmark"}], "name": "test1"}]}
Once the ID is matched with the name, Nova check information about the instance (I thought it was to validate if is in ACTIVE status, but i realized that even when is in STOPPED status the request is made it anyway).
GET http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee\
-H "User-Agent: python-novaclient" -H "Accept: application/json" \
-H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1"
RESP BODY: {"server": {"status": "ACTIVE", "updated": "2016-03-02T17:28:45Z", "hostId": "ca3a874dcad9079fcc6a0b10b0e2efaa394bc66b5335197fdd9c2498", "OS-EXT-SRV-ATTR:host": "liberty", "addresses": {"private": [{"OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:aa:1c:32", "version": 4, "addr": "10.0.0.6", "OS-EXT-IPS:type": "fixed"}]}, "links": [{"href": "http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "self"}, {"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "rel": "bookmark"}], "key_name": null, "image": {"id": "bf31eadd-c5f4-40f8-9ddb-30f688ca5e5f", "links": [{"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/images/bf31eadd-c5f4-40f8-9ddb-30f688ca5e5f", "rel": "bookmark"}]}, "OS-EXT-STS:task_state": null, "OS-EXT-STS:vm_state": "active", "OS-EXT-SRV-ATTR:instance_name": "instance-0000000a", "OS-SRV-USG:launched_at": "2016-03-02T17:28:45.000000", "OS-EXT-SRV-ATTR:hypervisor_hostname": "liberty", "flavor": {"id": "1", "links": [{"href": "http://192.168.200.208:8774/ee84411cdb8148d28674b129ef482f31/flavors/1", "rel": "bookmark"}]}, "id": "9165dbda-f54e-4186-b2cb-e6ca05ac53ee", "security_groups": [{"name": "default"}], "OS-SRV-USG:terminated_at": null, "OS-EXT-AZ:availability_zone": "nova", "user_id": "d9164a323be649c0a8c5c80fdd5bd585", "name": "test1", "created": "2016-03-02T17:28:34Z", "tenant_id": "ee84411cdb8148d28674b129ef482f31", "OS-DCF:diskConfig": "MANUAL", "os-extended-volumes:volumes_attached": [], "accessIPv4": "", "accessIPv6": "", "progress": 0, "OS-EXT-STS:power_state": 1, "config_drive": "", "metadata": {}}}
When we get the information, nova-api POST a request to nova-consoleauth for a VNC console.
POST http://192.168.200.208:8774/v2/ee84411cdb8148d28674b129ef482f31/servers/9165dbda-f54e-4186-b2cb-e6ca05ac53ee/action \
-H "User-Agent: python-novaclient" -H "Content-Type: application/json" \
-H "Accept: application/json" -H "X-Auth-Token: {SHA1}3b6262df9eaba5da33c1004805187806322201f1"\
-d '{"os-getVNCConsole": {"type": "novnc"}}'
DEBUG nova.api.openstack.wsgi [req-2201b9d6-5711-46d3-ac4d-669094f07527 \
d9164a323be649c0a8c5c80fdd5bd585 ee84411cdb8148d28674b129ef482f31 - - -] \
Action: 'action', calling method: , body: {"os-getVNCConsole": {"type": "novnc"}} \
_process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:789
Nova-consoleauth receives the console request and create an access URL while generates a temporary token for the vnc console.
INFO nova.consoleauth.manager [req-d4def6f9-1ab9-4626-b6a8-d81643ea5eb4 d9164a323be649c0a8c5c80fdd5bd585 ee84411cdb8148d28674b129ef482f31 - - -] \
Received Token: 3dfcd011-28f1-4cf3-8f5c-8cd18de4560e, \
{'instance_uuid': u'9165dbda-f54e-4186-b2cb-e6ca05ac53ee', \
'access_url': u'http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e',\
'token': u'3dfcd011-28f1-4cf3-8f5c-8cd18de4560e', 'last_activity_at': 1456940028.356214, \
'internal_access_path': None, 'console_type': u'novnc', 'host': u'liberty', 'port': u'5900'}
Nova-consoleauth answer to nova-api who also answers to the user with an access URL.
This URL got the following content on it:
- HTTP or HTTPS connection to nova-novncproxy IP
- Nova-novncproxy port
- A token to validate the VNC connection
RESP BODY: {"console": {"url": "http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e", "type": "novnc"}}
+-------+--------------------------------------------------------------------------------------+
| Type | Url |
+-------+--------------------------------------------------------------------------------------+
| novnc | http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e |
+-------+--------------------------------------------------------------------------------------+
Until now, nova-novncproxy service can be stopped or isn't used at all, is at this point the when proxy server enter into the game. The user connects through a web browser to the nova-novncproxy's URL provided by nova before.
DEBUG nova.console.websocketproxy [-] 192.168.200.1: \
new handler Process vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:828
Nova-vncproxy validate the issued token with the URL against nova-consoleauth.
nova.consoleauth.manager [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -] \
Checking Token: 3dfcd011-28f1-4cf3-8f5c-8cd18de4560e, True
When the token is validated, nova-novncproxy maps compute's node private IP (at this case port 5900) with the nova-novncproxy public IP(6080 port).
INFO nova.console.websocketproxy [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -]\
7: connect info: {u'instance_uuid': u'9165dbda-f54e-4186-b2cb-e6ca05ac53ee', u'\
internal_access_path': None, u'last_activity_at': 1456940028.356214, \
u'console_type': u'novnc', u'host': u'liberty', u'token': u'3dfcd011-28f1-4cf3-8f5c-8cd18de4560e', \
u'access_url': u'http://192.168.200.208:6080/vnc_auto.html?token=3dfcd011-28f1-4cf3-8f5c-8cd18de4560e'\
, u'port': u'5900'}
We can see how the python novncproxy process binds both IPs/port.
# ps aux | grep vnc
nova 14840 1.2 0.7 362096 41000 ? S 18:53 0:14 /usr/bin/python2 /usr/bin/nova-novncproxy --web /usr/share/novnc/
# netstat -putona | grep 14840
tcp 0 0 192.168.200.208:6080 192.168.200.1:59918 ESTABLISHED 14840/python2 keepalive (3,13/0/0)
tcp 0 0 192.168.122.73:57764 192.168.122.73:5900 ESTABLISHED 14840/python2 keepalive (3,13/0/0)
Nova-novncproxy starts the connection between the instance and user's browser session.
INFO nova.console.websocketproxy [req-399c7b58-700a-4779-b215-b12d10056813 - - - - -]\
7: connecting to: liberty:5900
Libvirt connects a vnc console into the instance, as we can see at the xml provided by virsh command. Also, port 5900 now is binded at qemu-kvm process.
# virsh dumpxml 2
...
<graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
...
# netstat -putona | grep 5900
tcp 0 0 0.0.0.0:5900 0.0.0.0:* LISTEN 5910/qemu-kvm off (0.00/0/0)
tcp 0 0 192.168.122.73:5900 192.168.122.73:57702 ESTABLISHED 5910/qemu-kvm off (0.00/0/0)
tcp 0 0 192.168.122.73:57702 192.168.122.73:5900 ESTABLISHED 11118/python2 keepalive (1,92/0/0)
Nova-novncproxy keeps the connection alive until browser session ends.
DEBUG nova.console.websocketproxy [-] \
Reaing zombies, active child count is 1 vmsg /usr/lib/python2.7/site-packages/websockify/websocket.py:828
When a token is not valid while authenticating against nova-consoleauth, we can see a message like the following.
INFO nova.console.websocketproxy [req-9164b32d-3ce1-441b-82c7-6c23c9a354d0 - - - - -] \
handler exception: The token '3dfcd011-28f1-4cf3-8f5c-8cd18de4560e' is invalid or has expired
Regards. Eduardo Gonzalez
Last modified 3yr ago