Fix deadlock in k8s provider

On a reasonably sized cluster:
63 nodes
87 services
90 endpoints

The initialization of the k8s provider would hang.

I tracked this down to the ResourceEventHandlerFuncs. Once you reach the
channel buffer size (10) the k8s Informer gets stuck. You can't read or
write messages to the channel anymore. I think this is probably a lock
issue somewhere in k8s but the more reasonable solution for the traefik
usecase is to just drop events when the queue is full since we only use
the events for signalling, not their content, thus dropping an event
doesn't matter.
This commit is contained in:
Phil Kates 2016-12-02 11:35:19 -08:00 committed by Yves Peter
parent 91d9b9811f
commit 87eac1dc1a

View file

@ -110,14 +110,22 @@ func (c *clientImpl) WatchIngresses(labelSelector labels.Selector, stopCh <-chan
return watchCh
}
func newResourceEventHandlerFuncs(events chan interface{}) cache.ResourceEventHandlerFuncs {
return cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) { events <- obj },
UpdateFunc: func(old, new interface{}) { events <- new },
DeleteFunc: func(obj interface{}) { events <- obj },
// eventHandlerFunc will pass the obj on to the events channel or drop it
// This is so passing the events along won't block in the case of high volume
// The events are only used for signalling anyway so dropping a few is ok
func eventHandlerFunc(events chan interface{}, obj interface{}) {
select {
case events <- obj:
default:
}
}
func newResourceEventHandlerFuncs(events chan interface{}) cache.ResourceEventHandlerFuncs {
return cache.ResourceEventHandlerFuncs{
AddFunc: func(obj interface{}) { eventHandlerFunc(events, obj) },
UpdateFunc: func(old, new interface{}) { eventHandlerFunc(events, new) },
DeleteFunc: func(obj interface{}) { eventHandlerFunc(events, obj) },
}
}
// GetService returns the named service from the named namespace