Android开发笔记[6]-离线中文TTS-526互联

摘要

在Android上实现离线中文TTS语音播报.

源码地址

[https://gitee.com/qsbye/AndTheStone/tree/compose]

Releasev0p1中有工程压缩包

平台信息

Android Studio: Electric Eel | 2022.1.1 Patch 2
Gradle:distributionUrl=https://services.gradle.org/distributions/gradle-7.5-bin.zip
jvmTarget = '1.8'
minSdk 21
targetSdk 33
compileSdk 33
开发语言:Kotlin,Java
ndkVersion = '25.2.9519653'

TTS技术简介

TTS技术，全称为文本到语音（Text-to-Speech）技术，是一种将文本转化为可听的语音输出的技术。它属于语音合成领域，可以将计算机自动生成的或外部输入的文字信息转变为流利的口语输出。

TTS技术在语音助理、智能音箱、导航系统、新闻播报等领域有广泛的应用。通过TTS技术，我们可以实现与计算机进行自然语言交互，使计算机能够以人类可理解的方式进行语音输出。

TTS技术的实现过程一般包括以下几个步骤：

文本分析：对输入的文本进行分析，提取语法词汇信息。
音素或音节生成：根据文本分析的结果，生成对应的语种、音素（或音节）信息。
韵律生成：根据文本的语义和语法信息，生成合适的韵律模式，使语音输出更加自然流畅。
合成语音：根据生成的音素（或音节）和韵律信息，将其转化为语音信号，实现文本到语音的转换。

TTS技术的发展离不开深度学习等技术的支持。近年来，深度学习方法在TTS领域取得了显著的进展，提高了语音合成的质量和自然度。

总之，TTS技术通过将文本转化为语音，实现了计算机与人类之间的语音交互，为人们提供了更加便捷和自然的人机交互方式。

基于TensorflowTTS的中文TTS

[https://github.com/benjaminwan/ChineseTtsTflite]
Android Chinese TTS Engine Base On Tensorflow TTS , use for TfLite Models Test。安卓离线中文TTS引擎，在TensorflowTTS基础上开发，用于TfLite模型测试。
可选两种模型：FastSpeech和Tacotron，这两种模型均来自TensorFlowTTS
文字转拼音方法来自：TensorflowTTS_chinese
因为是实时推理输出音频，故对设备性能有一定要求。
其中FastSpeech速度较快，但生成的音频拟人效果较差，可以用于普通中端以上手机。

实现

主要步骤

下载语音模型
[https://github.com/benjaminwan/ChineseTtsTflite/releases/tag/init]
models-tf.7z,models-tflite.7z,tensorflow-lite-2.8.0.aar,tensorflow-lite-select-tf-ops-2.8.0.aar
解压文件到目录

app/src/main/assets
│      baker_mapper.json
│      fastspeech2_quan.tflite
│      mb_melgan.tflite
│      tacotron2_quan.tflite

并把2个aar文件放到app/libs
3. build.gradle(app)

  //签名设置
    signingConfigs {
        debug {
            keyAlias 'androiddebugkey'
            keyPassword 'android'
            storeFile file("../appkey/debug.keystore")
        }
        tts {
            keyAlias 'chttstf'
            keyPassword 'chttstf'
            storeFile file('../appkey/chttstf.keystore')
            storePassword 'chttstf'
        }
    }

    dependencies {
    //中文TTS相关
    implementation fileTree(dir: "libs", include: ["*.jar", "*.aar"])
    implementation 'com.orhanobut:logger:2.2.0'
    implementation('org.tensorflow:tensorflow-lite-support:0.3.1') {
        exclude group: 'org.tensorflow', module: 'tensorflow-lite'
        exclude group: 'org.tensorflow', module: 'tensorflow-lite-api'
        exclude group: 'org.tensorflow', module: 'tensorflow-lite-select-tf-ops'
    }
    implementation 'com.belerweb:pinyin4j:2.5.1'
    implementation 'com.github.benjaminwan:MoshiUtils:1.0.6'
    implementation files('src/main/assets/usc_android_common_sdk/libs/usc.jar')
    }

移植[https://github.com/benjaminwan/ChineseTtsTflite/tree/main/app/src/main/java/com/benjaminwan/chinesettstflite]文件夹到工程的java/com目录
,注意在AndroidManifest.xml文件中注册Activity和Service,标记权限:

<application
        android:name="com.benjaminwan.chinesettstflite.app.App" />
        <!-- 省略 -->
<uses-permission android:name="android.permission.INTERNET"/>
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE"/>
<uses-permission android:name="android.permission.READ_CALENDAR"/>
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<service
android:name="com.benjaminwan.chinesettstflite.service.TtsService"
android:exported="true"
android:label="@string/app_name"
tools:ignore="ExportedService">
<intent-filter>
    <action android:name="android.intent.action.TTS_SERVICE" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
<meta-data
    android:name="android.speech.tts"
    android:resource="@xml/tts_engine" />
</service>
<activity
android:name="com.benjaminwan.chinesettstflite.ui.MainActivity"
android:configChanges="orientation|keyboardHidden|screenSize"
android:exported="true"
android:label="@string/title_activity_vision"
android:theme="@style/Theme.AndroidKotlinVirtualJoystick">
<intent-filter>
    <action android:name="android.speech.tts.engine.CONFIGURE_ENGINE" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>

<activity
android:name="com.benjaminwan.chinesettstflite.ui.DownloadVoiceData"
android:exported="true"
android:label="DownloadVoiceData">
<intent-filter>
    <action android:name="android.speech.tts.engine.INSTALL_TTS_DATA" />

    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>

<activity
android:name="com.benjaminwan.chinesettstflite.ui.CheckVoiceData"
android:exported="true"
android:label="CheckVoiceData">
<intent-filter>
    <action android:name="android.speech.tts.engine.CHECK_TTS_DATA" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>

<activity
android:name="com.benjaminwan.chinesettstflite.ui.GetSampleText"
android:exported="true"
android:label="GetSampleText"
android:theme="@android:style/Theme.Translucent.NoTitleBar" >
<intent-filter>
    <action android:name="android.speech.tts.engine.GET_SAMPLE_TEXT" />
    <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>

使用方式(必须在Activity中初始化用于TTS的viewModel)

class CameraActivity : AppCompatActivity() {

    //1. 暴露instance到外部
    companion object {
        lateinit var instance: CameraActivity
            private set
    }

    //2. 初始化用于TTS的viewModel
    val mainVM: MainViewModel by viewModels()
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        // 3. 将MainActivity实例分配给companion object的instance属性
        CameraActivity.instance = this

        //4. 正确初始化语音模型
        try{
            TtsManager.initModels(this@CameraActivity)}
        catch(e:Exception){
            Log.e("TTS","初始化模型错误")
        }

        //5. TTS播报
        cn.qsbye.Vision.CameraActivity.instance.mainVM.sayText("语音播放测试")
    }//end onCreate

}//end class

主要代码

TaskClass.kt

package cn.qsbye.Vision

import android.content.Context
import android.os.Bundle
import android.util.Log
import androidx.activity.viewModels
import androidx.appcompat.app.AppCompatActivity
import cn.qsbye.Vision.LogComponent
import com.benjaminwan.chinesettstflite.tts.TtsManager
import com.benjaminwan.chinesettstflite.ui.MainViewModel
import kotlinx.coroutines.*

/*
功能:
- 在这里编写任务(与串口交互,计算云台移动等)
*/

//实例化
val my_task = TaskClass()
var plane_num = 0

class TaskClass{
    val stages = listOf(
        "等待识别显示器边框",
        "阶段:云台参数修正",
        "正在识别\"开始测试\"",
        "阶段:基础题1",
        "基础题1:${plane_num}号飞机",
        "已锁定数字$plane_num",
        "正在识别\"开始测试\"",
        "阶段:基础题2",
        "基础题2:${plane_num}号飞机",
        "已锁定数字$plane_num",
        "正在识别\"开始测试\"",
        "阶段:基础题3",
        "基础题3:${plane_num}号飞机",
        "已锁定数字$plane_num",
        "正在识别\"开始学习\"",
        "阶段:发挥题1",
        "学习轨迹中",
        "正在识别\"开始测试\"",
        "正在识别\"开始学习\"",
        "阶段:发挥题2",
        "学习轨迹中",
        "正在识别\"开始测试\"",
        "阶段:发挥题3",
        "已完成全部任务"
    )

    //测试显示日志
    // 启动一个协程来每2秒更新 stages[i]
    fun test(context: Context) {
        GlobalScope.launch {
            var i = 0
            while (i < stages.size) {
                delay(3000) // 暂停2秒
                i++
                my_log.displayLogMessageTop(MyViewModel(),stages[i % stages.size]) // 显示更新后的 stages[i]
                //TTS播报状态
                cn.qsbye.Vision.CameraActivity.instance.mainVM.sayText(stages[i % stages.size])
            }
        }
    }//end test
}

CameraActivity.kt

package cn.qsbye.Vision

import android.Manifest
import android.content.Context
import android.os.Bundle
import android.util.Log
import androidx.activity.compose.setContent
import androidx.activity.viewModels
import androidx.appcompat.app.AppCompatActivity
import androidx.camera.core.CameraSelector
import androidx.camera.core.ImageAnalysis
import androidx.camera.core.Preview
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.camera.view.PreviewView
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
import androidx.compose.foundation.layout.fillMaxSize
import androidx.compose.foundation.layout.padding
import androidx.compose.material.Button
import androidx.compose.material.MaterialTheme
import androidx.compose.material.Surface
import androidx.compose.material.Text
import androidx.compose.runtime.*
import androidx.compose.runtime.snapshots.SnapshotStateList
import androidx.compose.ui.Modifier
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.Paint
import androidx.compose.ui.graphics.PaintingStyle
import androidx.compose.ui.graphics.drawscope.drawIntoCanvas
import androidx.compose.ui.graphics.nativeCanvas
import androidx.compose.ui.graphics.toArgb
import androidx.compose.ui.graphics.toComposeRect
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.platform.LocalLifecycleOwner
import androidx.compose.ui.unit.dp
import androidx.compose.ui.viewinterop.AndroidView
import androidx.core.content.ContextCompat
import cn.qsbye.Vision.theme.ComposePlaygroundTheme
import com.benjaminwan.chinesettstflite.tts.TtsManager
import com.benjaminwan.chinesettstflite.ui.MainActivity
import com.benjaminwan.chinesettstflite.ui.MainViewModel
import com.google.accompanist.permissions.ExperimentalPermissionsApi
import com.google.accompanist.permissions.PermissionRequired
import com.google.accompanist.permissions.rememberPermissionState
import com.google.mlkit.vision.objects.DetectedObject

@androidx.camera.core.ExperimentalGetImage
class CameraActivity : AppCompatActivity() {

    companion object {
        lateinit var instance: CameraActivity
            private set
    }

    //1. 初始化用于TTS的线程
    val mainVM: MainViewModel by viewModels()

    @ExperimentalPermissionsApi
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        //2. 保证正确加载model
        try{
            TtsManager.initModels(this@CameraActivity)}
        catch(e:Exception){
            Log.e("TTS","初始化模型错误")
        }

        // 将MainActivity实例分配给companion object的instance属性
        CameraActivity.instance = this

        //start Composal布局
        setContent {
            //自动配置浅色/深色主题
            ComposePlaygroundTheme {
                // A surface container using the 'background' color from the theme

                //Surface容器
                Surface(color = MaterialTheme.colors.background) {
                    val permission = Manifest.permission.CAMERA
                    val permissionState = rememberPermissionState(permission)

                    LaunchedEffect(Unit) {
                        permissionState.launchPermissionRequest()
                    }

                    //获取到摄像头权限
                    PermissionRequired(
                        permissionState = permissionState,
                        {}, {}, {
                            val detectedObjects = mutableStateListOf<DetectedObject>()

                            //Compose布局模式的Box自适应大小
                            Box {
                                //显示摄像头画面
                                CameraPreview(detectedObjects)

                                //start Canvas1:绘制检测到物体的画布层
                                //fillMaxSize:画布全屏
                                Canvas(modifier = Modifier.fillMaxSize()) {
                                    drawIntoCanvas { canvas ->
                                        detectedObjects.forEach {
                                            canvas.scale(size.width / 480, size.height / 640)

                                            //绘制矩形框
                                            canvas.drawRect(
                                                it.boundingBox.toComposeRect(),
                                                Paint().apply {
                                                    color = Color.Red
                                                    style = PaintingStyle.Stroke
                                                    strokeWidth = 5f
                                                })

                                            //绘制文字
                                            canvas.nativeCanvas.drawText(
                                                "TrackingId_${it.trackingId}",
                                                it.boundingBox.left.toFloat(),
                                                it.boundingBox.top.toFloat(),
                                                android.graphics.Paint().apply {
                                                    color = Color.Green.toArgb()
                                                    textSize = 20f
                                                })
                                        }
                                    }
                                }//end Canvas1

                                //start Canvas2: 人机交互图层
                                showCanvas2(this@CameraActivity)
                                //end Canvas2


                            }//end Box
                        }
                    )//end PermissionRequired

                }
            }
        }//end setContent
    }
}

@Composable
@androidx.camera.core.ExperimentalGetImage
private fun CameraPreview(detectedObjects: SnapshotStateList<DetectedObject>) {
    val lifecycleOwner = LocalLifecycleOwner.current
    val context = LocalContext.current
    val cameraProviderFuture = remember { ProcessCameraProvider.getInstance(context) }

    val coroutineScope = rememberCoroutineScope()
    val objectAnalyzer = remember { ObjectAnalyzer(coroutineScope, detectedObjects) }

    AndroidView(
        factory = { ctx ->
            val previewView = PreviewView(ctx)
            val executor = ContextCompat.getMainExecutor(ctx)

            val imageAnalyzer = ImageAnalysis.Builder()
                .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
                .build()
                .also {
                    it.setAnalyzer(executor, objectAnalyzer)
                }

            cameraProviderFuture.addListener({
                val cameraProvider = cameraProviderFuture.get()
                val preview = Preview.Builder().build().also {
                    it.setSurfaceProvider(previewView.surfaceProvider)
                }

                val cameraSelector = CameraSelector.Builder()
                    .requireLensFacing(CameraSelector.LENS_FACING_BACK)
                    .build()

                cameraProvider.unbindAll()
                cameraProvider.bindToLifecycle(
                    lifecycleOwner,
                    cameraSelector,
                    preview,
                    imageAnalyzer
                )
            }, executor)
            previewView
        },

        //填充全屏
        modifier = Modifier.fillMaxSize(),
    )

}

@Composable
fun showCanvas2(context: Context, viewModel: MyViewModel = MyViewModel()) {
    // 状态监听变量
    var myLogAreaTop: MutableState<String> = remember { mutableStateOf(my_log.logAreaTop) }
    var myLogArea: MutableState<String> = remember { mutableStateOf(my_log.logArea) }

    // 使用LaunchedEffect监听logAreaTop的变化并更新UI
    LaunchedEffect(myLogAreaTop.value) {
        myLogAreaTop.value = my_log.logAreaTop
    }

    // 使用LaunchedEffect监听logArea的变化并更新UI
    LaunchedEffect(myLogArea.value) {
        myLogArea.value = my_log.logArea
    }

    Canvas(modifier = Modifier.fillMaxSize()) {// 全屏显示

        drawIntoCanvas { canvas ->
            // 顶部横幅
            canvas.nativeCanvas.drawText(
                "${viewModel.myLogAreaTop}",
                10.0F,
                100.0F,
                android.graphics.Paint().apply {
                    color = Color.Red.toArgb()
                    textSize = 100f
                }
            )

            // 日志区
            val text = "${viewModel.myLogArea}"
            val paint = android.graphics.Paint().apply {
                color = Color.White.toArgb()
                textSize = 20f
            }
            val textBounds = android.graphics.Rect()
            paint.getTextBounds(text, 0, text.length, textBounds)
            val x = 10.0F // X坐标不变
            val y = canvas.nativeCanvas.height.toFloat() - 20.0F * 15 - 10.0F // 计算Y坐标
            canvas.nativeCanvas.drawText(text, x, y, paint)
        }//end drawIntoCanvas
    }

    // 刷新按钮
    Button(
        onClick = {
            // Update the state when the button is clicked
            myLogAreaTop.value = "${my_log.logAreaTop}"
            myLogArea.value = "${my_log.logArea}"
        },
        Modifier.padding(top = 100.dp)
    ) {
        Text("刷新")
    }

    // 测试横幅显示
    my_task.test(context)

}//end fun showCanvas2

效果

语音,不太好展示hhh