bing batching, yandex simplify, google fixes

CHANGELOG.md

@@ -3,10 +3,10 @@
  * FEATURE - Dynamic selection of translator during game session
  * FEATURE - Support BingTranslate API
  * FEATURE - Support LEC Offline Power Translator 15
- * FEATURE - Enable custom implementations of translators that can be loaded dynamically
- * FEATURE - Removed support for Excite translate because it only support the 'WWW' API in Unity due to missing TLS1.2 implementation
+ * FEATURE - Enable custom implementations of translators
+ * FEATURE - Removed support for Excite translate because it only support the 'WWW' API in Unity due to missing TLS1.2 support
  * FEATURE - Updated Watson translate to v3
- * FEATURE - Support for 'romaji' as output language. Only google supports this at the  moment
+ * FEATURE - Support for 'romaji' as output language. Only google supports this at the moment
  * BUG FIX - Too many small fixes to mention
  * MISC - {GameExeName} variable can now be used in configuration of directories and files
  * MISC - Changed the way the 'Custom' endpoint works. See README for more info

README.md

@@ -527,12 +527,12 @@ Follow these steps:
  2. Start a new project in Visual Studio 2017 or later. I recommend using the same name for your assembly/project as the "Id" you are going to use in your interface implementation. This makes it easier for users to know how to configure your translator
  3. Add a reference to the XUnity.AutoTranslator.Plugin.Core.dll that you downloaded in step 1
  4. You do not need to directly reference the UnityEngine.dll assembly. This is good, because you do not need to worry about which version of Unity is used.
-   * If you do need a reference to this assembly (because you need functionality from it) consider using an old version of it (if `UnityEngine.CoreModule.dll` exists in the Managed folder, it is not an old version!)
+    * If you do need a reference to this assembly (because you need functionality from it) consider using an old version of it (if `UnityEngine.CoreModule.dll` exists in the Managed folder, it is not an old version!)
  5. Create a new class that either:
-   * Implements the `ITranslateEndpoint` interface
-   * Inherits from the `HttpEndpoint` class
-   * Inherits from the `WwwEndpoint` class
-   * Inherits from the `ExtProtocolEndpoint` class
+    * Implements the `ITranslateEndpoint` interface
+    * Inherits from the `HttpEndpoint` class
+    * Inherits from the `WwwEndpoint` class
+    * Inherits from the `ExtProtocolEndpoint` class
 
 Here's an example that simply reverses the text and also reads some configuration from the configuration file the plugin uses:
 
@@ -612,28 +612,22 @@ internal class YandexTranslateEndpoint : HttpEndpoint
    {
       var data = context.Response.Data;
       var obj = JSON.Parse( data );
-      var lineBuilder = new StringBuilder( data.Length );
 
       var code = obj.AsObject[ "code" ].ToString();
       if( code != "200" ) context.Fail( "Received bad response code: " + code );
 
       var token = obj.AsObject[ "text" ].ToString();
-      token = JsonHelper.Unescape( token.Substring( 2, token.Length - 4 ) );
+      var translation = JsonHelper.Unescape( token.Substring( 2, token.Length - 4 ) );
 
-      if( string.IsNullOrEmpty( token ) ) return;
+      if( string.IsNullOrEmpty( translation ) ) context.Fail( "Received no translation." );
 
-      if( !lineBuilder.EndsWithWhitespaceOrNewline() ) lineBuilder.Append( "\n" );
-      lineBuilder.Append( token );
-
-      var translated = lineBuilder.ToString();
-
-      context.Complete( translated );
+      context.Complete( translation );
    }
 }
 ```
 
 This plugin extends from `HttpEndpoint`. Let's look at the three methods it overrides:
- * `Initialize` is used to read the API key the user has configured. In addition it calls `context.EnableSslFor( "translate.yandex.net" )` in order to disable the certificate check for this specific hostname. If this is neglected, SSL will fail in most versions of Unity. Finally, it throws an exception if the plugin cannot be used with the specified configuration.
+ * `Initialize` is used to read the API key the user has configured. In addition it calls `context.DisableCerfificateChecksFor( "translate.yandex.net" )` in order to disable the certificate check for this specific hostname. If this is neglected, SSL will fail in most versions of Unity. Finally, it throws an exception if the plugin cannot be used with the specified configuration.
  * `OnCreateRequest` is used to construct the `XUnityWebRequest` object that will be sent to the external endpoint. The call to `context.Complete( request )` specifies the request to use.
  * `OnExtractTranslation` is used to extract the text from the response returned from the web server.
 
@@ -643,10 +637,8 @@ After implementing the class, simply build the project and place the generated D
 
 **NOTE**: If you implement a class based on the `HttpEndpoint` and you get an error where the web request is never completed, then it is likely due to the web server requiring Tls1.2. Unity-mono has issues with this spec and it will cause the request to lock up forever. The only solutions to this for now are:
  * Disable SSL, if you can. There are many sitauations where it is simply not possible to do this because the web server will simply redirect back to the HTTPS endoint.
- * Use the `WwwEndpoint` instead.
-
-As mentioned earlier, you  can also use the abstract class `WwwEndpoint` to implement roughly the same thing. However, I do not recommend doing so, unless it is an authenticated service.
+ * Use the `WwwEndpoint` instead. I do not recommend using this base class unless it is an authenticated endpoint though.
 
 Another way to implement a translator is to implement the `ExtProtocolEndpoint` class. This can be used to delegate the actual translation logic to an external process. Currently there is no documentation on this, but you can take a look at the LEC implementation, which uses it.
 
-If instead, you use the interface, it is also possible to extend from MonoBehaviour to get access to all the normal lifecycle callbacks of Unity components.
+If instead, you use the interface directly, it is also possible to extend from MonoBehaviour to get access to all the normal lifecycle callbacks of Unity components.

src/Translators/BingTranslateLegitimate/BingTranslateLegitimateEndpoint.cs

@@ -7,6 +7,7 @@ using System.Net;
 using System.Reflection;
 using System.Text;
 using SimpleJSON;
+using XUnity.AutoTranslator.Plugin.Core;
 using XUnity.AutoTranslator.Plugin.Core.Configuration;
 using XUnity.AutoTranslator.Plugin.Core.Constants;
 using XUnity.AutoTranslator.Plugin.Core.Endpoints;
@@ -25,8 +26,7 @@ namespace BingTranslateLegitimate
       };
 
       private static readonly string HttpsServicePointTemplateUrl = "https://api.cognitive.microsofttranslator.com/translate?api-version=3.0&from={0}&to={1}";
-      private static readonly string RequestTemplate = "[{{\"Text\":\"{0}\"}}]";
-      private static readonly System.Random RandomNumbers = new System.Random();
+      private static readonly Random RandomNumbers = new Random();
 
       private static readonly string[] Accepts = new string[] { "application/json" };
       private static readonly string[] ContentTypes = new string[] { "application/json" };
@@ -40,6 +40,8 @@ namespace BingTranslateLegitimate
 
       public override string FriendlyName => "Bing Translator (Authenticated)";
 
+      public override int MaxTranslationsPerRequest => 10;
+
       public override void Initialize( IInitializationContext context )
       {
          _key = context.GetOrCreateSetting( "BingLegitimate", "OcpApimSubscriptionKey", "" );
@@ -54,10 +56,27 @@ namespace BingTranslateLegitimate
 
       public override void OnCreateRequest( IHttpRequestCreationContext context )
       {
+         StringBuilder data = new StringBuilder();
+         data.Append( "[" );
+         for( int i = 0 ; i < context.UntranslatedTexts.Length ; i++ )
+         {
+            var untranslatedText = JsonHelper.Escape( context.UntranslatedTexts[ i ] );
+            data.Append( "{\"Text\":\"" );
+            data.Append( untranslatedText );
+            data.Append( "\"}" );
+
+            if( context.UntranslatedTexts.Length - 1 != i )
+            {
+               data.Append( "," );
+            }
+         }
+         data.Append( "]" );
+
+
          var request = new XUnityWebRequest(
             "POST",
             string.Format( HttpsServicePointTemplateUrl, context.SourceLanguage, context.DestinationLanguage ),
-            string.Format( RequestTemplate, JsonHelper.Escape( context.UntranslatedText ) ) );
+            data.ToString() );
 
          if( Accept != null )
          {
@@ -74,14 +93,18 @@ namespace BingTranslateLegitimate
 
       public override void OnExtractTranslation( IHttpTranslationExtractionContext context )
       {
-         var arr = JSON.Parse( context.Response.Data );
+         var arr = JSON.Parse( context.Response.Data ).AsArray;
 
-         var token = arr.AsArray[ 0 ]?.AsObject[ "translations" ]?.AsArray[ 0 ]?.AsObject[ "text" ]?.ToString();
-         token = JsonHelper.Unescape( token.Substring( 1, token.Length - 2 ) );
+         var translations = new List<string>();
+         for( int i = 0 ; i < arr.Count ; i++ )
+         {
+            var token = arr[ i ].AsObject[ "translations" ]?.AsArray[ 0 ]?.AsObject[ "text" ]?.ToString();
+            var translation = JsonHelper.Unescape( token.Substring( 1, token.Length - 2 ) );
 
-         var translated = token;
+            translations.Add( translation );
+         }
 
-         context.Complete( translated );
+         context.Complete( translations.ToArray() );
       }
    }
 }

src/Translators/GoogleTranslate/GoogleTranslateEndpoint.cs

@@ -45,6 +45,7 @@ namespace GoogleTranslate
       private bool _hasSetup = false;
       private long m = 427761;
       private long s = 1179739010;
+      private int _translationsPerRequest = 10;
 
       public GoogleTranslateEndpoint()
       {
@@ -55,12 +56,17 @@ namespace GoogleTranslate
 
       public override string FriendlyName => "Google! Translate";
 
-      public override int MaxTranslationsPerRequest => 10;
+      public override int MaxTranslationsPerRequest => _translationsPerRequest;
 
       public override void Initialize( IInitializationContext context )
       {
          context.DisableCerfificateChecksFor( "translate.google.com", "translate.googleapis.com" );
 
+         if( context.DestinationLanguage == "romaji" )
+         {
+            _translationsPerRequest = 1;
+         }
+
          if( !SupportedLanguages.Contains( context.SourceLanguage ) ) throw new Exception( $"The source language '{context.SourceLanguage}' is not supported." );
          if( !SupportedLanguages.Contains( context.DestinationLanguage ) ) throw new Exception( $"The destination language '{context.DestinationLanguage}' is not supported." );
       }

src/Translators/YandexTranslate/YandexTranslateEndpoint.cs

@@ -58,22 +58,16 @@ namespace YandexTranslate
       {
          var data = context.Response.Data;
          var obj = JSON.Parse( data );
-         var lineBuilder = new StringBuilder( data.Length );
 
          var code = obj.AsObject[ "code" ].ToString();
          if( code != "200" ) context.Fail( "Received bad response code: " + code );
 
          var token = obj.AsObject[ "text" ].ToString();
-         token = JsonHelper.Unescape( token.Substring( 2, token.Length - 4 ) );
+         var translation = JsonHelper.Unescape( token.Substring( 2, token.Length - 4 ) );
 
-         if( string.IsNullOrEmpty( token ) ) return;
+         if( string.IsNullOrEmpty( translation ) ) context.Fail( "Received no translation." );
 
-         if( !lineBuilder.EndsWithWhitespaceOrNewline() ) lineBuilder.Append( "\n" );
-         lineBuilder.Append( token );
-
-         var translated = lineBuilder.ToString();
-
-         context.Complete( translated );
+         context.Complete( translation );
       }
    }
 }

src/XUnity.AutoTranslator.Plugin.Core/AutoTranslationPlugin.cs

@@ -1938,6 +1938,15 @@ namespace XUnity.AutoTranslator.Plugin.Core
          onContinue();
       }
 
+      private IEnumerator EnableBatchingAfterDelay()
+      {
+         yield return new WaitForSeconds( 60 );
+
+         _batchLogicHasFailed = false;
+
+         XuaLogger.Current.Error( "Re-enabled batching." );
+      }
+
       void Awake()
       {
          if( !_initialized )
@@ -2101,7 +2110,6 @@ namespace XUnity.AutoTranslator.Plugin.Core
                if( _endpoint.IsBusy ) break;
 
                var kvps = _unstartedJobs.Take( _endpoint.Endpoint.MaxTranslationsPerRequest ).ToList();
-               //var batch = new TranslationBatch();
                var untranslatedTexts = new List<string>();
                var jobs = new List<TranslationJob>();
 
@@ -2144,8 +2152,6 @@ namespace XUnity.AutoTranslator.Plugin.Core
             {
                if( _endpoint.IsBusy ) break;
 
-               // ERROR ERROR ERROR --- MEGA BUG MEGA BUG, MUST REMOVE FROM _unstartedJobs INSIDE LOOP!!!!
-
                var kvp = _unstartedJobs.FirstOrDefault();
 
                var key = kvp.Key;
@@ -2204,7 +2210,11 @@ namespace XUnity.AutoTranslator.Plugin.Core
          }
          else
          {
-            // might as well re-add all translation jobs, and never do this again!
+            if( !_batchLogicHasFailed )
+            {
+               StartCoroutine( EnableBatchingAfterDelay() );
+            }
+
             _batchLogicHasFailed = true;
             for( int i = 0 ; i < jobs.Length ; i++ )
             {
@@ -2301,6 +2311,11 @@ namespace XUnity.AutoTranslator.Plugin.Core
          }
          else
          {
+            if( !_batchLogicHasFailed )
+            {
+               StartCoroutine( EnableBatchingAfterDelay() );
+            }
+
             _batchLogicHasFailed = true;
             for( int i = 0 ; i < jobs.Length ; i++ )
             {

src/XUnity.AutoTranslator.Plugin.Core/Configuration/Settings.cs

@@ -35,7 +35,7 @@ namespace XUnity.AutoTranslator.Plugin.Core.Configuration
       public static bool IsShutdown = false;
       public static bool IsShutdownFatal = false;
       public static int TranslationCount = 0;
-      public static int MaxAvailableBatchOperations = 40;
+      public static int MaxAvailableBatchOperations = 50;
 
       public static readonly float MaxTranslationsQueuedPerSecond = 5;
       public static readonly int MaxSecondsAboveTranslationThreshold = 30;

src/XUnity.AutoTranslator.Plugin.Core/Endpoints/Http/HttpEndpoint.cs

@@ -79,11 +79,11 @@ namespace XUnity.AutoTranslator.Plugin.Core.Endpoints.Http
          // prepare request
          OnCreateRequest( httpContext );
          if( httpContext.Request == null ) httpContext.Fail( "No request object was provided by the translator.", null );
-
+         
          // execute request
          var client = new XUnityWebClient();
          var response = client.Send( httpContext.Request );
-
+         
          // wait for completion
          var iterator = response.GetSupportedEnumerator();
          while( iterator.MoveNext() ) yield return iterator.Current;