我们都知道在分布式日志当中,exceptionless客户端是把日志写到
Elasticsearch数据库,就像我们把数据写入到关系数据库一样;
既然是写入,那么在短时间大数据量的情况下,写入就会涉及到效率的问题;
首先我们看下 exceptionless源码
public class InMemoryObjectStorage : IObjectStorage {private readonly Dictionary<string, Tuple<ObjectInfo, object>> _storage = new Dictionary<string, Tuple<ObjectInfo, object>>(StringComparer.OrdinalIgnoreCase);private readonly object _lock = new object();public InMemoryObjectStorage() : this(1000) {}public InMemoryObjectStorage(int maxObjects) {MaxObjects = maxObjects;}public long MaxObjects { get; set; }public int Count {get { return _storage.Count; }}
我们写入到日志的方法 ,首先会把日志写到内存当中;但是为了防止日志过大,导致服务器的内存被过度消耗的问题 ,这边是限制了 日志的条数的 1000;
当有新的日志进来时,如果是超过1000,那么旧的日志是会被抛弃的;
public bool SaveObject<T>(string path, T value) where T : class {if (String.IsNullOrWhiteSpace(path))throw new ArgumentNullException("path");lock (_lock) {_storage[path] = Tuple.Create(new ObjectInfo {Created = DateTime.Now,Modified = DateTime.Now,Path = path}, (object)value);if (_storage.Count > MaxObjects)_storage.Remove(_storage.OrderByDescending(kvp => kvp.Value.Item1.Created).First().Key);}return true;}
这显然不是 我们想要的,我们需要的是完整的日志;
public SubmissionResponse PostEvents(IEnumerable<Event> events, ExceptionlessConfiguration config, IJsonSerializer serializer) {if (!config.IsValid)return new SubmissionResponse(500, message: "Invalid client configuration settings");string data = serializer.Serialize(events);string url = String.Format("{0}/events", GetServiceEndPoint(config));HttpResponseMessage response;try {HttpContent content = new StringContent(data, Encoding.UTF8, "application/json");// don't compress data smaller than 4kbif (data.Length > 1024 * 4)content = new GzipContent(content);_client.Value.AddAuthorizationHeader(config.ApiKey);response = _client.Value.PostAsync(url, content).ConfigureAwait(false).GetAwaiter().GetResult();} catch (Exception ex) {return new SubmissionResponse(500, exception: ex);}int settingsVersion;if (Int32.TryParse(GetSettingsVersionHeader(response.Headers), out settingsVersion))SettingsManager.CheckVersion(settingsVersion, config);return new SubmissionResponse((int)response.StatusCode, GetResponseMessage(response));}
所以我们需要在PostEvents 这个方法做文章;
目前我们的做法是:
就是把日志直接写到消息队列,然后做一个消费日志的订阅队列的服务,从队列读取数据,在写到Elasticsearch;